Creating a Performance Engineering Statement of Work is a very complex process. I hope this blog makes it easier to really dig deep/ask the right questions to assess both the application and business requirements. One of the largest variables in a SOW is time – how much time will it take to complete a project.
Both the client and the engineers need to remain flexible and keep the communication flowing, especially during challenges. For example, the client has a requirement for a “basic” load script which exercises certain web/mobile application features. Further investigation might reveal that this feature is a challenge to automate. The engineer needs to communicate the technical findings and the client needs to decide whether to keep that transaction in the mix or append the original SOW to add more work. Automating complex transactions that involve dynamic content and complex logic can take sometimes many hours (if not a couple days) of development work.
By asking the below questions to the client, you are better prepared to create a more accurate SOW for any load testing project. Please share with your colleagues!
- What is the total number of different load scripts which your application requires to create a realistic production workload? Each script represents a specific user persona type. For example: a Researcher, a Shopper, a Video Watcher. Please limit each script to 7 business transactions including Login/Logout.
- What is the expected peak load of concurrent users? This is the highest amount of users expected to be actively using your web or mobile application. You can derive this number from either business analysts or from data analytics of your existing deployed production application.
- What are the transactional throughput targets of your business transactions? For example, Do you expect 100,000 “submits” per 1 hour?
- Is the target application a web or mobile application? Or both? Do you have both a web site and a native mobile application for the same purpose?
- What are the communication protocols of this application? Just http/s? Or AMF, Flex, RTMP, HLS, Websocket? Please inquire with your development team if unsure of this answer.
- If this is an API load test, can development provide sample requests as well as sample successful and failure responses?
- What types of devices or browsers need to be emulated during this load test?
- From what geographical locations do you want to distribute the load of virtual users? Is your application a global or only available in certain countries?
- Do you want to emulate networks with bandwidth restrictions?
- Is this an independent application or are there 3rd party integrations? Are those 3rd party integrations to be excluded or included in the load testing?
- Is this load test to be executed in a production or testing environment? If a test environment, does the deployment mimic production in hardware, configuration and code releases?
- Does the database need to be reset in between each test execution?
- Are the workflows of the transactions of the scripts sequential steps or do they require more complex logic such as loops, if/then’s, conditional statements?
- Is there dynamic content which needs to be handled/correlated? Please provide detail.
- Do the users need to login/authenticate? If so, is there a pool of test user accounts prepared for these tests?
- Does the application allow the same userID login multiple times/multiple active sessions?
- How many test executions would you like to schedule? Please note, a methodical approach is to validate scalability limitations is recommended. Encourage at least 2-3 tests per workload. Also, please include baseline tests (low load tests) to execute before peak load tests.
- What is the duration of each test? Please note, we recommend longer test durations for high volumes of users. For example, for high volume tests, a ramp up schedule of 10000 virtual users per hour.
- What other types of tests are required: Ramping to target production load, Stress (beyond production load), Longevity (12 + hours)?
- Is the application deployment being monitored by New Relic, AppDynamics, CloudWatch or another monitoring solution?
- Will the supporting application team be available to monitor the application during any of the tests?
- Does this engagement require the load tool to have embedded infrastructure monitoring?
- Do you prefer to use and open soure or commercial load tool? Please note, choose the appropriate tool based on it’s time saving features, not just on price.
Sometimes even the most accurate of SOW’s will experience creep depending on the challenges which are encountered. Remain in the state of mind of a problem solver and work together to eliminate those hurdles, one by one, until the goals are accomplished.