Load Impact Load Testing and Performance Testing

Do the old (1.x) test configurations work?

Test configurations from Load Impact 1.x have been migrated to Load Impact 2.0. Unfortunately, the way tests are run in 2.0 is different from in 1.x so you can’t run your old test configurations as they are.

In Load Impact 1.x we executed tests in a way we call “client level execution”, which means that the test was configured to start at a certain load level, say 10 concurrent clients loading the target site, and then increase the load in steps up to a specified cut-off point where the tests end. This could mean that a test might start at 10 clients, run for a while, shut down all the 10 clients and then start 20 clients instead, run for a while, shut down the 20 clients and then start 30 clients instead, and so on.

With Load Impact 2.0, however, we introduce “continuous execution” where we don’t “restart” the test at each configured load level – instead we keep the previously started clients and just add more clients in order to increase the load, and we add those clients slowly, over time. This means that a test that is configured to e.g. ramp up from 0 to 50 clients in 5 minutes will add 10 clients per minute, or 1 client every 6 seconds. The tests always start at 1 client and then ramp up slowly over time, and running clients are never shut down until the whole test ends (well, if you ramp down the load at any point during the test, clients will be shut down, of course).

Continuous tests are more time-efficient and offer a more realistic ramp up scenario than the client-level tests do, so we have decided to stop using the old client-level tests and just offer the continuous mode in Load Impact 2.0. This means that you will not be able to execute your old test configurations. However, you can easily create new test configurations and you are also able to easily reuse your old load scripts, as the latter have been transformed into user scenarios for you in Load Impact 2.0.

Do the simulated clients in a test cache objects that are loaded?

No, the simulated clients in a test will never cache anything (except for cookies, which is a special case and described below). This means that in a test, every client that loads a page from your site will behave like a new visitor to the site and thus be quite “heavy” on the server.

Note, though, that because most major browsers have very low default settings for the maximum size of the browser cache, more users than you might think will be hitting your site with an empty cache – see this blog post for more info.

So how does client-side caching work?

Normally, what happens is that a client (browser) loads a certain resource from the server – say a .jpg image – that it needs in order to render a certain web page. A while later, the user opens another page where the same image is used. The client realizes it has the image cached on the user’s hard drive, but it needs to make sure the image hasn’t been updated on the server since last time it was fetched, so the client issues a normal request for the same image file again, but adds a special header – the If-Modified-Since: header, that tells the server that it only needs to send the resource (the image) back to the client if the image has been updated since the last time the client fetched it.

And how does caching work with Load Impact?

When a Load Impact load test is executed, it will start a certain number of simulated clients/users that access your site simultaneously. Usually, the number of clients is increased (ramped up) over time, thus increasing the load on the server. Every simulated client will execute the load script that is associated with the particular user scenario the client is running. When the client has finished executing its load script, it will execute it again, and again, and again – an infinite number of times until the load test is finished.

While a client runs its load script, cookies set by the server (via the “Set-Cookie:” header) will be reused by the client in subsequent requests. All up until the load script execution ends – then all cookies will be deleted before the next execution of the script. However, cookies, or more specifically the “Cookie:” header, is currently the only header that is set automatically by the client. Other headers, such as the “If-Modified-Since:” header will not be set unless the user specifies it in the load script, and this is why caching is not emulated automatically.

So how can you emulate client-side caching?

Currently, the only good way to do it is to edit your load script and specifically make your requests use the “If-Modified-Since:” header. Another option would be to remove some requests completely from the load script (i.e. the second time a certain .jpg file is loaded in the same load script you can just remove that request from the script) but this would mean the server gets fewer requests than it would normally get. The server would then never get the request that includes the “If-Modified-Since:” header. While the server would likely not send anything back to the client if/when it gets a request with that header set, every request still generates a little bit of work for the server, so removing the requests completely is not optimal (though it might be an acceptable workaround in many cases).

How does the text editor work?

Load scripts are used to program the behaviour of simulated clients in a load test. Each user scenario has its own load script that determines exactly what objects the simulated user will load from the target site during the load test.

The load scripts are programs written using the high-level programming language Lua and can be very simple or very complex, depending on the requirements and programming skills of the tester.

There are two different editors available to load script programmers – the graphical editor and the text editor. The graphical editor is menu-based and makes coding a bit easier if you’re not a very strong programmer or not too familiar with Lua, while the text editor is more powerful and easier to use for the advanced programmer.

The text editor allows you to type in Lua code directly, designing your load script from scratch. It also lets you perform on-screen cut-and-paste, which can be useful as it lets you write the code somewhere else, in an editor of your choice.

Load script example

 -- (This is a comment line)
 -- My load script V1.0
 --
 -- Make an initial HTTP request to get the index HTML page from the site
 http.request_batch({
    {"GET", "http://loadimpact.com/", nil}
 })
 --
 -- Make a series of requests to fetch remaining objects on page
 http.request_batch({
    {"GET", "http://loadimpact.com/favicon.ico", nil}, 
    {"GET", "http://loadimpact.com/css/all.css", nil},
    {"GET", "http://loadimpact.com/js/jquery.js", nil},
    {"GET", "http://loadimpact.com/js/loadimpact.js", nil},
    {"GET", "http://loadimpact.com/js/main.js", nil},
    {"GET", "http://loadimpact.com/js/browserDetect.js", nil},
    {"GET", "http://loadimpact.com/js/cachemanager.js", nil},
    {"GET", "http://loadimpact.com/images/rss-16.png", nil},
 })
 --
 -- Sleep a random amount of time between 10 and 20 seconds (simulates user think time)
 client.sleep(math.random(10000, 20000), 1000)

How do I configure my proxy settings for the Session Recorder?

In order to use our Session Recorder, you will need to configure your browser settings. You also need to make sure you do not have any cached files for the site you are about to record. When you have configured your browser, test your proxy settings to ensure that the settings are properly configured, then click on “Start Recording”.
Note: Remember to clear your cache once this is done. Steps written below.

—- FOR MAC USERS —-
Configuring your proxy settings for Chrome (Mac)
Step 1
Select “Preferences”
Step 2
Click “Under the Hood” and select the “Change Proxy Settings” option. You should get the proxy dialogue.
Step 3
In the window that appears, check the “Web Proxy (HTTP)” and Secure Web Proxy (HTTPS)” options.
Step 4
In the field of “Web Proxy Server” enter “proxy.loadimpact.com”
(Note: You have to do this for BOTH the Web Proxy and Secure Web Proxy options)
Step 5
In the “Port” field, enter the 5 digit port number provided
Step 6
Confirm your changes by pressing “Ok” in both dialogs.

Configuring your proxy settings for Safari (Mac)
Step 1
Select Safari’s “Preferences”
Step 2
Click on the “Advanced” tab and “Change Settings” with the Proxies label. You should get the proxy dialogue.
Step 3
In the window that appears, check the “Web Proxy (HTTP)” and Secure Web Proxy (HTTPS)” options.
Step 4
In the field of “Web Proxy Server” enter “proxy.loadimpact.com”
(Note: You have to do this for BOTH the Web Proxy and Secure Web Proxy options)
Step 5
In the “Port” field, enter the 5 digit port number provided
Step 6
Confirm your changes by pressing “Ok” in both dialogs.

—- FOR WINDOWS USERS —-
Configuring your proxy settings for Chrome (Windows)
Step 1
Select “Options” in the Toolbar Menu>“Under The Hood”>“Change Proxy Settings”>“LAN Settings”.
Step 2
Enter “proxy.loadimpact.com” as address.
Step 3
In the port field, enter the 5 digit port number provided.
Step 4
Confirm your changes.

Configuring your proxy settings for Firefox
Step 1
Open the “Tools” menu and select “Options”.
Step 2
Click “Advanced” and select the “Network” tab.
Step 3
Under the “Connection” label, click the “Settings” button.
Step 4
In the window that appears, select “Manual proxy configuration”.
Step 5
In the field of “HTTP proxy” enter “proxy.loadimpact.com”.
Step 6
In the “Port” field, enter the 5 digit port number provided
Step 7
Confirm your changes by pressing “Ok” in both dialogs.

Configuring your proxy settings for Internet Explorer
Step 1
Open the “Tools” menu and select “Internet Options”.
Step 2
Select the “Connections” tab and press the “LAN Settings” button.
Step 3
Under the “Proxy server” settings, check the box labeled “Use a proxy server for your LAN”.
Step 4
In the “Address” field enter “proxy.loadimpact.com”.
Step 5
In the “Port” field, enter the 5 digit port number provided.
Step 6
Confirm your changes by pressing “Ok” in both dialogs.

—- CLEAR YOUR CACHE —-
(Essential for Session Recorder to record all requests, ensuring a proper load test is performed)
Clear cache
Step 1
Open the settings for “Temporary internet files”.
Step 2
Clear the cached files.
Final Step
Click “Go” on this page and record your session

How do I create new result metrics?

Load Impact collects a lot of different result metrics by default, but sometimes you might want to measure something specific that we don’t currently include in the standard suite of metrics. This is where our custom metrics functionality comes in handy.

Custom metrics are very easy to use. Basically, you just calculate a sample point for your metric, then you call the result.custom_metric() function to store and aggregate the sample. You can store as many samples as you like, but if more than one sample is stored in the same 3-second period, an average value will be computed and stored instead of the two (or more) individual values.

Example:

 -- Measure the time it takes to complete several HTTP requests, 
 -- and then store that time as the result metric "total_load_time"
 start_time = util.time()
 http.request_batch({
    { "GET", "http://loadimpact.com", nil, { ["X-Myheader"]="Something" } },
    { "GET", "http://loadimpact.com/image1.jpg" },
    { "POST", "http://loadimpact.com/login_ajax.php", nil, { ["X-Anotherheader"]="Something Else" }, "login=test&password=test" }
 })
 end_time = util.time()
 result.custom_metric("total_load_time", end_time - start_time)

All metrics that you collect yourself in this way will of course be available on the test report page, allowing you to plot graphs of your metrics etc. just as with the built-in metrics.

How do I print debug messages from a script?

The Log module provides you with functionality to log and display arbitrary (small) text messages during script execution. This is useful while debugging a load script, but also if you want to e.g. verify content that is returned from a server and get an alert if some content is not what you expected. All log messages are displayed at the very bottom of the test result page. Of course, if you use the “validate” functionality to validate a user scenario, you never visit the test result page and in that case your log messages will be displayed directly in the validation window instead.

There are several log functions that perform the same, basic action (log text) but that formats the text slightly differently. You have:

Example:

-- Make an HTTP request and verify returned content. If it is not what we expect
-- we log an error using the log.error() function
local request = http.get(
    "http://test.loadimpact.com/pi.php?decimals=18",
    nil, 
    nil, 
    nil, 
    nil, 
    100  -- Tell the system it should store up to 100 bytes of body data for us
 )
 if request.body ~= "3.141592653589793238" then
   log.error(
      "PI calculator returned unexpected result when stressed with "
         .. test.get_clients() .. " simulated clients: " .. request.body
   )
 end

Limitations

In order to prevent excessive logging by a malfunctioning or badly programmed script, we limit the number of messages that can be logged in any given 3-second reporting time period (see - What result metrics are reported for a URL? for more info about how reporting data is stored and transferred) to 100 messages. This means that if your script logs 1000 messages within 3 seconds, chances are you will only see the first 100 of those messages. The rest will be quietly discarded by the load generator application.

Each log message can also be max 1024 bytes (characters) in length before it is truncated. I.e. if you try to log a text message that is 1100 bytes long, the actual text that is logged will only be the first 1024 bytes of your message, the remaining 76 bytes will be lost.

Earlier, (before May 2012) logging worked in a different way – we used fixed-size, rotating buffers to store log messages, which meant that in case you logged too many messages in a 3-second reporting period, the first messages would be lost, not the last ones. Also, log messages were not transferred to, and stored in, the results database immediately. They were collected by the load generator application in these rotating buffers instead. Then, at the end of every 3-second interval the rotating buffers were flushed and all logged messages were sent to the results database to be stored (meaning they can then be seen by the user in the test result output). The problem with this approach was that if you e.g. used the logging functionality to debug runtime problems in your load script – for example printing out the value of a variable just before the script crashes – there was a very big chance you would never see the log output as the next line of code will usually be executed within milliseconds of your logging statement being executed, meaning your logged message never gets the chance to be sent to the results database.

How do I test a site where you need to login?

Load Impact supports both basic HTTP authentication and HTTP POST operations, as well as HTTPS. This means we support most common methods for logging in to sites.

Login via HTTP POST

Many sites let users login by filling in their username and password in input fields of a form that is then POST:ed to the web server. Load Impact supports HTTP POST operations, which mean that you may edit your load script and provide the login credentials (username/password) in a POST operation. When the server gets the right login credentials through this POST operation, it sets a session cookie in your browser. The session cookie is used for all future communications with the web server, authenticating you as a logged-in user. Load Impact automatically sets cookies when asked to by the server, so accessing pages as a logged-in user works fine if you just make sure your load script performs that initial POST operation.

More information:

• POST requests
• The Load Impact load script API – http.post

Basic HTTP Authentication

Basic HTTP Authentication is a method where the server will only honor requests if they contain a special HTTP header (the “Authorization:” header) where the client has to place the login credentials (username and password) that are to be used to gain access to the site or page.

Load Impact supports Basic HTTP Authentication by the use of URLs in this format:

     http://username:password@www.mydomain.com/index.php

(We also allow HTTP request headers to be specified for each transaction, so it is possible to create the data for the “Authorization:” header in the load script code and then add the header to each request, but that is more work, of course)

More information:

• Basic HTTP Authentication
• Setting HTTP headers for a request
• The Load Impact load script API

How do I use parameterized data?

Data stores are used to parameterize data in load scripts. You create a data store from a CSV file containing the data you want to use in your load scripts. The CSV file should contain rows with one or more column fields in each row. The column fields should be separated by some separator character or string. You upload your CSV file in the user scenario configuration interface, associating the data store with that user scenario. In this process you also give the data store a name.

Inside your load script you will then be able to use the datastore module functionality to initialize a datastore object by using the datastore.open() and referring to the name of your data store. The datastore object, once initialized, can be used to extract data from your data store.

Data stores are a very convenient way to deal with large amounts of data you might need to access from your load script. One of the most common use cases is that you might have a list with 10,000 usernames and passwords that you want the simulated clients in a load test to use to access restricted content on your site. Entering all these usernames and passwords by hand into your load script would be a time-consuming task, but you might be able to export them to a file in CSV format, allowing you to upload them and create a data store from them.

Example:

Let’s say you have a CSV file called “user_credentials.csv” that looks like this:

 username,password
 john,topsecret
 sally,verysecret
 anne,alsosecret

It contains the usernames and passwords of three users, and you want to use these in your load test. You would then open the user scenario configuration interface, and click on the “+” icon next to the text that says “Selected data store: None”. Here is a screenshot of the relevant part of the screen:

After clicking the “+” icon, the load script code section at the bottom will be hidden and you will instead be shown the data store configuration section that looks like this:

Here you would usually get a list of data stores to choose from, but in this case we have no existing data stores, so the only thing we can do is create a new one by clicking the “New data store” button. That will take you to the data store upload interface that looks like this:

Now you can click on the “Choose file” button. This will let you choose a file on your local computer that you want to upload. This file should, as described a bit earlier here, be a text file in the CSV format. It can be up to a maximum of 50 MB in size. In our example we select our small “user_credentials.csv” file that contains only four rows. Next, we will see this:

Note that the file has not been uploaded yet, but parsed on the client side, via Javascript. The Javascript has found out that the file is 62 bytes in size, and it gives you a preview of the first few lines of data from the file. In our case, the file is so small that the preview shows all of it, but most of the time your CSV files are probably going to be quite large. The preview shows you what data your file contains, and also how it is going to be parsed when it is uploaded. On the screenshot above you can see that we will get four rows, with two columns in each row. However, we are not so interested in including the first row from the file – the one that says “username” and “password”. Those are just the column names, not actual login credentials that we can use. So we want to get rid of them, and we can do that by clicking the “+” icon next to the text “File parsing options”. This is what we get then:

Note that with these extra options you can change the field seperator character to be something other than the comma character, for example it is quite common that CSV files have semicolon as field separator. You can also specify what character is used to quote string values inside the CSV file (“Text delimiter”). Finally, you also have an option to start parsing the CSV file at some specific row (“Use data from line”). This setting defaults to line 1 – i.e. start using data from the first line of the CSV file. In our case we want to start using data from line #2 of the CSV file, and in the screenshot above we have just changed this parameter to “2”, which is why the preview now only shows the last three lines from the file.

When we are done setting file parsing options we click the “Upload” button, which starts the upload of the CSV file. If the file is large, the upload may take a little while. When the upload is done, the system will start converting the data into an internal format so it can be used in load tests. This can also take a little while (up to several minutes) if the file is large. When everything is done you will see a new data store object available to your user scenario:

Now, you just have to click the “Select” button to associate your data store with the current user scenario, then you can close the data store selection interface by clicking the “-” icon next to the “Selected data store” text. That will return you to the standard user scenario configuration interface, allowing you to start writing load script code to use data from your data store.

In your load script, you could then use the following code:

 -- Initialize data store
 local ds = datastore.open("My Data Store")
 -- Get a random row from data store
 local row = ds:get_random()
 -- The row is a table (array). Extract username field from it
 local username = row[1]
 -- ...and password field
 local password = row[2]
 -- Issue a HTTP POST request to login this user on our site
 http.post("http://my.dom.ain/login.aspx", nil, nil, 
               "username=" .. username .. "&password=" .. password)

What are VU and SBU users?

When executing a load test we will simulate a certain number of users accessing your web site concurrently. There are, however, two different types of simulated users you can choose from, and they behave differently.

First, there is the Virtual User (or VU, for short). This is the default type of user, and the simplest type. A Virtual User will only use a single network connection when loading resources from a target host. Resources can be loaded from multiple hosts in the same test, however, which will then result in one connection per target host, per VU.

Then there is the Simulated Browser User (SBU). Thisuser type can use multiple concurrent network connections when loading resources from a single target host. This means that if e.g. the simulated user needs to fetch four different resources from the same target host, it can open four concurrent network connections and transfer all four resources in parallel. Obviously, this results in faster page loads, but also a lot more stress on the target server that has to serve four connections at once instead of just one.

Any load testing tool that can just simulate the VU type of user will not be very accurate in predicting how many concurrent users the target site can handle, because in real life, a web browser will practically always use multiple connections when fetching things from a server, and that puts a lot more stress on the server. We therefore provide the option to use either VU or SBU user types. However, a test using the SBU user type will cost more credits because simulating it consumes more infrastructure resources (memory, CPU and bandwidth) on our side aswell. If you go to the test configuration screen you can switch between VU and SBU users in the load test execution plan and see how big the price difference is for your particular test, depending on what user type you choose to simulate.

When choosing to use SBU users in your test you are also able to have your SBU users emulate a wide range of popular browsers such as IE, Firefox, Chrome, Safari, Opera etc. and even different versions of those browsers. The performance impact of emulating different browser can be huge – for instance, IE6 will never open more than two concurrent network connections to your webserver, while most newer browsers will open up to six connections.

Tip: Go to browserscope.org if you want to learn more about unique browser behaviour.

What IP addresses do Load Impact use to generate load?

Some people have firewalls that prevent the general public out on the Internet from reaching their staging web sites. They often configure their firewalls to block traffic from all IP addresses except the ones used by their developers. To run tests with Load Impact they therefore need to add the Load Impact load generator IP addresses to the list of addresses that are allowed to pass the firewall.

Load Impact 2.0 generates load solely from the cloud, using Amazon cloud servers. However, you will need to include our main loadimpact.com IP addresses as well.

The main loadimpact.com IP addresses:

195.178.177.176/29
109.228.153.0/24

IP addresses for Load Impact 2.0 cloud generator nodes:

US East (Northern Virginia):

72.44.32.0/19 (72.44.32.0 – 72.44.63.255)
67.202.0.0/18 (67.202.0.0 – 67.202.63.255)
75.101.128.0/17 (75.101.128.0 – 75.101.255.255)
174.129.0.0/16 (174.129.0.0 – 174.129.255.255)
204.236.192.0/18 (204.236.192.0 – 204.236.255.255)
184.73.0.0/16 (184.73.0.0 – 184.73.255.255)
184.72.128.0/17 (184.72.128.0 – 184.72.255.255)
184.72.64.0/18 (184.72.64.0 – 184.72.127.255)
50.16.0.0/15 (50.16.0.0 – 50.17.255.255)
50.19.0.0/16 (50.19.0.0 – 50.19.255.255)
107.20.0.0/15 (107.20.0.0 – 107.21.255.255)
107.22.0.0/16 (107.22.0.0 – 107.22.255.255)
23.20.0.0/14 (23.20.0.0 – 23.23.255.255)

US West (Northern California):

204.236.128.0/18 (204.236.128.0 – 204.236.191.255)
184.72.0.0/18 (184.72.0.0 – 184.72.63.255)
50.18.0.0/16 (50.18.0.0 – 50.18.255.255)
184.169.128.0/17 (184.160.128.0 – 184.169.255.255)

US West (Oregon):

50.112.0.0/16 (50.112.0.0 – 50.112.255.255)

EU (Ireland):

79.125.0.0/17 (79.125.0.0 – 79.125.127.255)
46.51.128.0/18 (46.51.128.0 – 46.51.191.255)
46.51.192.0/20 (46.51.192.0 – 46.51.207.255)
46.137.0.0/17 (46.137.0.0 – 46.137.127.255)
46.137.128.0/18 (46.137.128.0 – 46.137.191.255)
176.34.128.0/17 (176.34.128.0 – 176.34.255.255)
176.34.64.0/18 (176.34.64.0 – 176.34.127.255)

Asia Pacific (Singapore)

175.41.128.0/18 (175.41.128.0 – 175.41.191.255)
122.248.192.0/18 (122.248.192.0 – 122.248.255.255)
46.137.192.0/18 (46.137.192.0 – 46.137.255.255)
46.51.216.0/21 (46.51.216.0 – 46.51.223.255)

Asia Pacific (Tokyo)

175.41.192.0/18 (175.41.192.0 – 175.41.255.255)
46.51.224.0/19 (46.51.224.0 – 46.51.255.255)
176.32.64.0/19 (176.32.64.0 – 176.32.95.255)
103.4.8.0/21 (103.4.8.0 – 103.4.15.255)
176.34.0.0/18 (176.34.0.0 – 176.34.63.255)

South America (Sao Paulo)

177.71.128.0/17 (177.71.128.0 – 177.71.255.255)

IP addresses updated and accurate as of 27 Jan 2012. Please refer to Amazon Forums
for the updated list of IP addresses.

What is a test configuration?

A test configuration describes the load test you want to run – how many simulated users the test should load your site with, and what those users should do on your site (what pages they should load on the site).

You can create and edit test configurations on the test configurations page.

In the test configuration you have to supply a target URL to the site you want to test. If you then try to start the test without performing any additional configuration, Load Impact will start an automatic test for you, running a ramp-up test at a quite low load level (0-50 virtual users).

If you want to run a test with higher (or lower) load levels, you have to expand the Load test execution plan configuration section when editing your test. This section will allow you to specify what load levels the test will use, and how quickly to ramp up or down to/from those load levels. You can also specify whether the test should use virtual users or simulated browser users, if you want.

Finally, you can configure your test to use one or more user scenarios also. When editing your test configuration you will see a User scenarios section that can be expanded by clicking on its plus (+) icon. In this section you can specify that a certain percentage of all the simulated users in the load test will run a certain user scenario. You can also choose which geographic region to run each scenario from, allowing you to distribute the simulated load across different types of users coming from different geographic locations.

What is a the Session Recorder and how does it work?

Our Session Recorder is a nifty tool (Edit Test> New Test> Advanced> Record Session> Proxy Recorder) that is available for all to use. What this tool does is that it opens your URL in a new tab on your browser, and records all actions taken on the page. All GET requests are captured, and a script is then automatically generated for you at the end of the session.

How to use our Session Recorder
Select the “User Scenarios” Tab
Click on “Create User Scenario”
Select either our graphical editor (Basic) or text editor (Advanced)
Enter your target URL and click “Start Recording”

A new window with your URL will then pop up, and whatever actions taken on this page will be recorded. Once the user actions are completed, you can close the window and stop the recording, and the script will automatically be generated for you. You can then choose to further edit the script if you’re into advanced scripting, and run the tests accordingly.
Note: Remember to change back the settings in your browser once your Session Recording is over, so as to not use the proxy anymore.

For more information on how to configure your proxy settings for the Session Recorder, please refer here.

What is a user scenario?

A user scenario is more or less a synonym for a “load script” in Load Impact 1. The user scenario defines what URLs or web pages should be loaded by the simulated clients during a load test.

You can create multiple user scenarios and use them all in a single load test. E.g. if you have a shopping site, you can then have one user scenario that simulates a user just browsing on the site, and another user scenario that simulates a user who logs in on the site, puts some products into the shopping cart, and then performs a check-out.

User scenarios can be created in several ways. Starting on the User scenarios page, you click the Create scenario button to create a new scenario. After that you have several options – if you know exactly what individual URLs you want the user scenario to load, you can write the load script for the scenario manually. This is rarely the case though, and it is a lot of manual work, so a better option is to enter a Script generation URL and then either click the Analyze button or the Start recorder button.

The Analyze button will make the system load the Script generation URL and parse it like a web browser does. I.e. if you specified the address of a certain web page, the system would load that URL, parse the HTML code it gets and find out what other resources (images, CSS files, etc) would be necessary to load for a web browser to be able to render the web page. Then a load script would be generated that fetched all these resources, in the same order a web browser would have fetched them. When the user scenario is run in a load test it would be just like a real user loaded that one, single web page with all its dependent resources.

The Start recorder button will make Load Impact launch a HTTP proxy server, allowing you to configure your web browser so that it uses this proxy server to access the web. Then a new browser window will be opened for you and the Script generation URL page will be loaded in that window. You will be able to surf around on your site, loading multiple pages, and when you are done you can just close the new window and click the Stop recorder button in the old window. This will create a load script that contains all the HTTP transactions that were executed by your own browser while you were surfing around in the new browser window, mimicking your behaviour exactly (including pauses in between page loads).

Tip: You don’t have to configure user scenarios in order to run a load test – we will create a user scenario automatically for you if you don’t specify one. The automatic user scenario will be based on the target URL you type in for your test, and will load just that one page.

Where do you generate the load/traffic from?

Load Impact uses the Amazon public cloud to generate load test traffic. This means that we support traffic generation from all the major Amazon AWS data centers around the world. An updated list of these locations can be found here:

http://aws.amazon.com/about-aws/globalinfrastructure/

At the time of writing (March 2012), the following locations are available for load generation:

• Ashburn, Virginia, US
• Palo Alto, California, US
• Portland, Oregon, US
• Dublin, Ireland
• Singapore
• Tokyo, Japan
• Sâo Paulo, Brazil

How do I interpret test results?

Interpreting results from a load test is not easy. It is almost as difficult as configuring a load test. When you run a load test using Load Impact, you will always get a default results graph on the view test page. This graph will give you a very rough estimate of how much time, as experienced by a user, it would take to load your web page (or pages, if you configured your test to access multiple pages) when your site was accessed by a certain number of concurrent users.

In most situations, it is correct to say that If you get a flat or falling graph, it means your site can handle the load comfortably and if you want to know where its limits are, you need to increase the load level.

Apart from that, you have to look at the absolute load time values reported and decide what are acceptable values for your site. Maybe your particular application or your particular users will balk at using your site/application if the page load time exceeds 10 seconds, or maybe they will do so at 5 seconds. Only you know your own users. Usually, it is worthwhile to decide on a number and then load your site until you reach that number, because then you know how many users you can handle with acceptably low load times.

Note that it is very important to know how your users behave on your site, in order to be able to create a realistic load test. What pages they visit and how long they stay on those pages will greatly affect how many concurrent users your site can handle, and if your load test does not correctly emulate the behaviour of your real users, the results will not tell you how many real users your site can handle. You should start by using a tool like Google analytics to learn more about the behaviour of your users, and then create Load Impact user scenarios to emulate how your users behave on your site.

Metrics reported in a test

User load time is the “default” metric that gets shown in the default chart for every test started, along with the number of active clients. Below is an example of the default chart, showing user load time as a blue graph and active clients as a green graph (note that colors may vary if you create your own, custom graphs).

The user load time metric reports the amount of time a simulated client has spent loading resources from the tested server. If the load test is well prepared and executed, this metric will report the actual load time as experienced by a typical user on your site, at different load levels. However, “well prepared and executed” in this case means you have to:

• Know what client applications (browsers) are most commonly used when users access your site
• Know what the most common user behaviours on your site are – what do people do on the site, exactly?
• Know where your users are most commonly located geographically (or even better, how much network delay there is between them and your site)
• Accurately simulate these behaviours/circumstances through correct configuration of your load test and your user scenarios

Obviously, there has to be a large amount of luck involved to get these things right through just entering the site URL and pressing a “start” button, and that is why many such tests do not provide very accurate estimates of how many real-world users the site can handle. The test case is simply not realistic enough. An unrealistic test case can nevertheless provide you with valuable information. You can run such a test in order to learn where bottlenecks are, you can also run the test before and after optimization efforts, in order to learn what actions give you the most performance improvements. Load testing is rarely a waste of time, even if very little effort is spent on preparations, but it is useful to know the limitations of your testing.

To sum up, user load time tells you how much time it took the simulated clients to perform all the HTTP transactions in your user scenario. It measures actual time passed fetching HTTP objects from the server, and ignores time spent executing e.g. sleep() statements or similar. It also takes into account the speedup generated when fetching multiple things in parallel, like all modern browsers do. This means that if you run a test using SBU users, and set the right number of max connections for those users, the user load time metric will give you a fairly accurate measurement of how long it would have taken a real user, using a real browser, to load the same content.

Another standard metric is accumulated load time. This metric is the sum total of all individual load times, for all objects loaded by the client. It is a metric you can use to compare web server performance between load tests, as it gives a good picture of how fast things are loading in general, but it does not tell you “page load time” as experienced by a user. Mainly because a user will normally run a browser that loads several things in parallel, over multiple TCP connections and the actual time passed will then not be a sum of all individual load times. An example:

1. A real user runs Firefox 3.5 and navigates to a web page – http://www.dom.ain
2. Firefox opens a single connection to the www.dom.ain server and requests the main HTML document “/”
3. After 0.2 seconds, Firefox has retrieved the main HTML document. It parses it and notices references to a CSS file (“main.css”), two images (“image1.jpg” and “image2.jpg”) and a javascript (“jquery.js”)
4. Firefox now requests “main.css” over the connection it already has open, then it opens three more connections to the server and requests “image1.jpg”, “image2.jpg” and “jquery.js” over those connections.
5. 0.3 seconds later, Firefox has received the “main.css” file
6. 0.1 seconds later it has received the “image1.jpg” file and the “jquery.js” file
7. 0.1 seconds later it has received the “image2.jpg file”

Note that in order to understand these examples, and how web performance is affected depending on the user’s choice of browser, it is important to know a bit about networking and how web servers and clients make use of the connection-oriented/stateful TCP procotol to transfer data.

Total (real) time elapsed is 0.2 + 0.3 + 0.1 + 0.1 = 0.7 seconds. This is what the user load time metric would report if this had been a load test – the actual time passed as experienced by the user. The accumulated load time metric, however, will report 0.2 + 0.3 + 0.4 + 0.4 + 0.5 = 1.8 seconds. These are the individual load times for all objects/resources loaded. The main HTML file took 0.2 seconds to load, the “jquery.js” took 0.3 seconds, the “image1.jpg” and “image2.jpg” took 0.4 seconds (remember that we started loading them at the same time as we started loading “jquery.js”, in step #4 above) and “image2.jpg” took a total of 0.5 seconds to load.

Some common graph types

1. The flat graph

The chart below shows a green graph that displays number of active clients in a test, and a blue graph that shows user load time (response time) during the test. Despite increasing the load (the green graph) from 1 to 25 concurrent clients we can see that the blue graph doesn’t change – it fluctuates slightly up and down but stays at the same average level regardless of the load level. This tells us that the system we are testing is nowhere near stressed by the load we are subjecting it to. We probably need to use substantially higher load levels to see any kind of performance degradation.

2. The rising graph

Here we have a test where the load level is ramped up from about 75 to 500 clients over the course of about 12 minutes (the test started at 0 clients but we have zoomed in the graph to show the most interesting parts). We can see that the blue graph showing response time is flat for a while, then it starts to increase at approx. 200 concurrent clients, and continues increasing linearly throughout the test.

From this graph we can deduce that demand for some resource is exceeding availability of that resource around 200 clients, causing a graceful performance degradation from that point onwards as every client has to share the scarce resource equally. Shortage of some system resources (e.g. network bandwidth) will cause this kind of graceful/linear performance degradation, while others will result in very fast, exponential performance degradation (system memory shortage, for instance). Note that graceful degradation usually becomes exponential once the shortage gets severe enough.

3. The exponential graph

This graph looks flat to begin with, but then it starts to rise, and keeps rising faster and faster as load increases. This is called exponential behaviour, where every little bit of load that is added will cause an ever larger performance degradation. When you see this type of graph you know that you are not far away from the extreme limit of your system’s performance.

Note that because the charts are stretched out horizontally it can sometimes be hard to see the “exponentiality” of a graph. Below you can see a screenshot of the same graph as the one above (not covering the exact same time range though), after clicking the “switch view” button that causes the chart to be only half the horizontal size:

What does the map show?

The map shows the geographical location of the target system and the load generators used in the test.

On the map, the green lines visualize traffic flows – thicker lines mean more traffic. The lines are updated dynamically as traffic flows change during the test.

What is the CSV export format?

Test results data can be exported as CSV files (comma-separated values), for easy importing into e.g. Excel or other spreadsheet applications. Below is a description of the CSV format. Let’s start with a couple of examples.

Example 1
a line logging the current number of active clients (4) that run the user scenario “Citerus JS-test” on the Ashburn load zone:

2011-11-22 16:13:51,__li_clients_active,,"Amazon US East (Ashburn, US)",Citerus JS-test (ID: 1005974),,,,,,4.0

Example 2
a line logging min/max/avg transaction times and number of transactions for the www.dom.ain/jquery.validate.js URL by clients running the user scenario “Citerus JS-test” on the Ashburn load zone and where the HTTP status (response) code for the transactions was 200:

2011-11-22 @16:13:51,__li_url_9fab2690b88546b3addf566e19af9dd9_200,http://www.dom.ain/jquery.validate.js,“Amazon US East (Ashburn, US)”,Citerus JS-test (ID: 1005974),200,58.82,178.21,85.68,112,

Detailed information about the CSV fields

All lines use the following column format (fields):

Time,Result ID,URL,Load zone,User scenario,Status code,RTT min,RTT max,RTT avg,Count,Value

Note some string values have comma characters in them. Like “Amazon US East (Ashburn, US)”. These strings will be enclosed in quotation marks (”“) to make the string be parsed as one single field value, despite containing the separator character (comma).

What metrics are reported for a URL?

For each URL in a test, we report what status codes the tested server has responded with, and for each of those status codes, we report the max, min and average reponse time, and the number of times we have seen that particular status code.

On the test results page, there is an expandable section labeled “URLs” where you see a list of every URL included in the test, and some statistics about the URL. This is how it looks:

More about results metrics

Response times, and other metrics, are measured continually throughout the test, in 3-second intervals. The load generator will record HTTP transaction times for each individual URL and every 3 seconds it reports the maximum, minimum and average transaction time seen for that URL during the most recent 3-second interval.

These values are stored in the results database, and downloaded by a client-side Javascript when you open your test result (or, in the case of a running test, results are downloaded incrementally as they are being generated). The Javascript code in your browser has access to all the individual data samples, but the graph component will aggregate data when necessary, in order to generate decent-looking graphs. The aggregation level depends on what zoom level you set your chart at. The chart can display up to about 170 points in a graph, which means that a 5-minute test (300 seconds) that has about 100 individual data samples (one sample every 3 seconds) can be shown in its entirety with no aggregation, while a 10-minute test will not quite fit into the 170 points but has to be slightly aggregated.

Currently these metrics are stored once every 3 seconds during a test, and can be graphed:

What status codes are there?

Load Impact stores the HTTP status/return codes on a per-URL basis, and reports for each individual URL how many times it has seen a particular status code from the server, and what response times (avg/min/max) it got for that particular code.

An example: Imagine a load test ramping up from 1 to 5000 concurrent users that contains a user scenario that loads “http://some.dom.ain/index.html”. At the start, when the load level is low, the web server being tested performs as it should and returns HTTP 200 codes plus the HTML file that was requested, but when the load level passes 4000 concurrent users, the server starts to run out of some resource (e.g. database connections) and begins returning “500 server error” instead of the expected content. Towards the end of the test the 500-codes (errors) occur more and more frequently. This means that we might end up with a test result where the “index.html” resource got e.g. 23,567 responses with the status code “200” and 12,856 responses where the status code was “500” and no content was returned. Most of the 200-codes occured early in the test while the 500-codes occured in the later stages, when the load level was higher.

If you want to see how fast the server is delivering content at various points during the test you can choose to plot the average transaction response time for the 200-codes. If you want to see the error rate throughout the test you can plot the “count” (number of times a response code was seen) metric for the 500-codes (perhaps combine it with the “count” value for the 200-codes in the same chart and you can see both the successful and the failed transactions next to eachother).

For more info on HTTP status codes, see http://en.wikipedia.org/wiki/List_of_HTTP_status_codes

There are also some 4-digit (non-HTTP) status codes that can be found in test results. They are related to the lower level networking functionality. Here is a list of the most common ones:

Why is the load time I am experiencing different from the test result?

Sometimes users load the same site that they are load testing and experience a much faster load time then what is reported by our system. This could be due to a few possible reasons:

1) Client-side (browser) caching caused resources not to be loaded. Our load generators do not simulate caching. All simulated clients appear as “new users” to the server. Read more about client-side caching in Load Impact

2) User is on a network that is geographically closer to the server. Our load generators are located in Ashburn (US), Dublin (Ireland), Palo Alto (US), Portland (US), Singapore and Tokyo (Japan). If your server is located far away from our load generators but close to you, traffic will have to take a longer route to reach our load generator. The extra network delay introduced can affect test results greatly, depending on the composition of your site (many small files will result in bad performance over high-latency network links). A longer route also makes it more likely that there are bandwidth bottlenecks somewhere along the way.

3) The web page can appear fully loaded graphically to the user but there might still be HTTP transactions executing in the background. Our load generators report load times that take into account all HTTP transactions on the website (as scripted in the user scenario).

4) The load test uses the VU user type which is not as efficient when loading web pages as a modern browser is. Configure your test to use the SBU user type if you want a more realistic simulation of a modern browser. Read more about VU and SBU users

How can I get an invoice for the purchase?

To get a invoice for your purchase you can click on “Credits” after logging in.
Scroll down to the “Payments” category.
Click the “View receipt” button.

You can choose to either save or print the receipt

I’m having problems with my payment

We accept credit card payments using VISA, Mastercard and AMEX. If you experience problems with payment, it may be due to a variety of factors. We urge you to get in touch with us if you can’t seem to pay with your credit card, as we are interested to know what the most common problems are.

Common problems include:

- Mistyping the credit card number, your name, card expiry date or your security code.

There is a lot of information you have to give the payment service processor in order to make a payment, and it is easy to mistype something somewhere.

- 3Dsecure problems.

3Dsecure is a new security feature that when enabled, forces credit card holders to use a password when they make purchases on the Internet. If the payment page asks you for a password, it is probably because of this feature. Here is a question from a user:

“This could not be a bug, but when i try to make a purchase with my Australian mastercard or visa, it prompts me to enter my online banking password to authenticate the payment. This is very odd as even though the site is secure, i refuse to enter my online password to internet banking on a non banking site. I cant seem to find the link between paying with my credit card (CC) online and internet banking. Most of the time when i use my CC online, the payment is made and thats it. do you provide other payment options, even paypal as i really like to use your system but unable to use it. Any help or explanation would be much appreciated”

And a response from Jonas at Load Impact:

“The password requested is not the password to your bank, but rather a password you set at your bank. This is an extra security feature, called 3-D Secure, used by Visa and MasterCard for online credit card payments. You should also be able to see a phrase, also set at your bank, that verifies you are actually communicating with your bank rather than a scam site.”

You can read more about 3-D Secure at Wikipedia: http://en.wikipedia.org/wiki/3-D_Secure
Your bank can tell you more about 3-D Secure as well.

- The risk check fails, causing the purchase to be denied.

Whenever a purchase is made, there is a risk calculation made to determine how (un)likely it is that the transaction is fraudulent or unauthorized. Credit card companies don’t tell us exactly what parameters they look at when making these risk calculations, but it involves things such as recent transaction history etc. I.e. if someones credit card is used to make 1,000 purchases in a single day, that might be seen as extraordinary behaviour and result in future transactions on the same card being denied.

What are credits used for?

Credits is the virtual currency used to execute load tests. Whenever you’re logged on to your loadimpact.com account and execute a load test you will use some credits. Exactly how many depends on the size of the load test, and how long the test is running.

Free credits, and free testing:

When registering a new user account you automatically get 5 credits for free. If you use up these credits, you will get new ones each month (a refill). This means that as long as you only run small-scale tests, and not too often, your testing will not cost you anything. Owning an account is always free

If you want to buy more credits, what do they cost?

The base cost of 1 credit is $3. If you have no credits at all, and try to run a load test that will use up 10 credits, the system will tell you that you need more credits, and offer to let you purchase the 10 credits you need for $30. This means that you never have to buy anything from us until the very moment you want to run your load test – you can always pay “per test”.

Credit packages:

If you want to save money, paying per test is not as economical as buying one of our Credit packages. These packages allow you to buy credits in bulk, at a discount. The discount gets higher the more credits you buy at the same time. See our pricing page for more information about packages.

But you can run tests without even registering an account, can’t you?

Yes, when you are not logged on, you can always run anonymous tests from the loadimpact.com start page. These tests never use up any credits. However, results from anonymous tests are indexed by search engines and publicly displayed for all to see. As they are fully automatic, they can also not be configured in any way – you can’t decide exactly what pages will be loaded during the test or what load levels will be used.

What is your refund policy?

We will honor all refund requests. If you feel you haven’t got your money’s worth when using Load Impact, or if you made a mistake when you purchased some credits, you will get your money back. We want every customer to feel safe about buying from us.

How do you prevent abuse of the service?

Load Impact might be used by someone to run illegal load tests on a site, effectively causing a denial-of-service attack on the site in question. We implement a number of measures to make our service unattractive to potential attackers. These measures include, but are not limited to:

• Anonymous tests are limited in size and duration
  The anonymous tests ramp up from 1 to 50 VUs during a 5-minute period. The maximum load level at the end of the 5 minutes will be equal to about 15 real users accessing the site simultaneously.

• Byte and transaction counters
  We remember every byte transferred to/from sites (IP addresses) we test, and for anonymous tests we allow only a certain number of tests, bytes or transactions to be executed for a particular target site per 48 hours. This means that an abuser will not be able to run an arbitrary number of anonymous load tests for a particular site. They will be limited to running (small-scale, see above) tests for a total period of 10-15 minutes every 48 hours. This means they will be able to test a site less than 1% of the time, worst case.

• Black listing
  We continuously monitor the system for abusive behaviour, and if we see anyone testing a site excessively, we will black list the site being tested so noone can test it. We adopt a shoot-first-ask-questions-later policy when it comes to black listing. We will also black list email providers if we detect that people using that provider are registering multiple accounts with us just to gain free credits.

We used to require a loadimpact.txt file placed in the root directory of a site in order to run large-scale tests against the site. This requirement has been dropped as we saw that it added a lot of problems for people who were running legitimate tests. We will, however, make it possible in the future for people to use loadimpact.txt and/or DNS to themselves manage black- or white listing of their site.

We welcome people to contact us in case they notice any abusive behaviour, which will be swiftly dealt with.