Requires Decipher Cloud
SST stands for Survey Stress Testing
The command line tool,
tsst can be used to test survey validity and performance. The tool simulates a virtual respondent that randomly chooses answers at each question.
When running the
tsst command, you'll see something similar to:
[proj1234]$ tsst . 100 Assuming . refers to selfserve/9d3/proj1234 Loading survey selfserve/9d3/proj1234. Survey changed since last SST run; clearing out old data first (data.clear) Running test 97/100: ************************************************.. All done. 100/100 succeeded. CPU usage per page: 9 This survey has enough progress data to go live.
Before you can set a survey LIVE, you must obtain at least 100 qualified completes using SST.
Running SST will clear survey data if a change has been made to your survey.
2: Usage & Syntax
tsst tool can be configured in several ways to change the way it processes your survey.
tsst [-q] [-v] [-n] [-N] [-o] SURVEY #_OF_ATTEMPTS
SST Optional Configurations
tsst modifiers are optional, but sometimes useful when your survey has complex logic.
This is considered cheating. The preferred method is to create a good
||No clear data||False||Use
Custom styles are ignored by default.
Developers only: use
This feature is rarely necessary. You should not increase value beyond 4.
tsst -qv . 100 will run 100 SST attempts without terminating (hopefully) and ignoring <validate> elements
tsst -qvoNn . 500 will run 500 SST attempts without terminating (hopefully), ignoring <validate> elements, attempting to skip optional questions, without clearing current data and ignoring question styles
3: Specifying Answers
Use a file named
sst.txt in your project's directory to manually declare values for questions and variables in your survey.
If your SST attempts result in more terminates than qualified completes, then the
sst.txt file can help provide valid data for those questions with complex logic.
<number label="Q1" size="3" verify="range(1, 125)"> <title>Please enter your age below.</title> </number> <suspend/> <term cond="not Q1.check('18-64')">Q1: Age less than 18 or greater than 64</term>
In the code above, any value that is not within the range 18 - 64 will result in termination. To avoid wasting SST attempts on terminates, we can manually specify the value(s) to use at question Q1 in the file
Q1: 18 24 34 44 54 64
Now SST will only use the values provided and supply answers that will avoid the termination logic altogether.
Here are some other examples:
list: 1 2 3 # specify values for extraVariables (e.g. list) co: us fr br # specify values for extraVariables (e.g. co) Q1: c1 c2 # specify columns to select for a radio question Q2: r1 r2 r3 # specify rows to select for a radio question Q3.r1: c1 c2 c3 # specify columns to select for a specific row in a radio question Q4.c2: r1 r2 # specify rows to select for a specific column in a radio question Q5.r1: ch1 ch2 # specify choices to select for a specific row in a select question Q6.c1: 100% # specify which column to select in a checkbox question Q7.r1: 50% # r1 at Q6 will be checked 50% of the time Q7.r2.c1: 100% # c1 will be checked for r2 at Q6 100% of the time Q8: 5 10 20-50 # specify numbers or number ranges for number questions timeSpent: 50-200 # override the default 30s per question returned
For more information, check out Specifying Answers for Simulated Data (SST).
3.1: Skipping an Element
You can specify to skip any element with the SST attribute. This prevents the survey stress tester from testing that element. When
sst="0" is set, the SST system ignores the element.
<number label="Q1" size="3" verify="range(1, 125)" sst="0"> <title>Please enter your age below.</title> </number>
<term cond="not Q1.check('18-64')" sst="0">Q1: Age less than 18 or greater than 64</term>
For more information, check out Element Attributes.
4: Additional Components
When SST is ran, the following components of your project are affected:
4.1: Test Data History & Report
A full report of all SST attempts is stored at /admin/sst/list. You can access your project's test data report using the following link (update the highlighted part with your project):
The test data report includes:
- a list of all SST sessions initiated
- the number of SST attempts per session
- the status of each attempt (finished, fatal, fail, loop)
- fatal error message
- survey view of the pages seen
- question coverage
- page time and traffic load
For more information about the test data report, see Simulated Data History Report.
4.2: Coverage Report
The Coverage Report shows how many questions have been seen. You can access your project's coverage report in the data downloads menu beneath the Direct to Browser column by selecting Coverage.
You should use this tool to verify that there are no errors in logic. If there are questions in your survey that have been seen few or 0 times, this may indicate an error.
To account for complex logic, you many need to configure the project's
sst.txt file to simulate the correct answers in order to traverse those complex paths. Learn more.
4.3: Progress Bar
The progress bar relies on simulated data. In order for the percentage to line up correctly, it's suggested that you run at least 200 SST attempts through the final version of your survey. This helps establish all of the possible paths that a respondent can take.
If your survey is especially complex, consider running more SST to account for those complex paths.