Zu Hauptinhalten wechseln

FV Decipher Unterstützung

Alle Themen, Ressourcen für FV Decipher benötigt.

 
decipher

Running Simulated Data from the Command Line

  Requires Decipher Cloud

1:  Overview

SST stands for Survey Stress Testing

The command line tool, tsst can be used to test survey validity and performance. The tool simulates a virtual respondent that randomly chooses answers at each question.

When running the tsst command, you'll see something similar to:

[proj1234]$ tsst . 100
Assuming . refers to selfserve/9d3/proj1234
Loading survey selfserve/9d3/proj1234.
Survey changed since last SST run; clearing out old data first (data.clear)
Running test 97/100: ************************************************..


All done. 100/100 succeeded. CPU usage per page: 9
This survey has enough progress data to go live.

Before you can set a survey LIVE, you must obtain at least 100 qualified completes using SST.

Running SST will clear survey data if a change has been made to your survey.

2:  Usage & Syntax

The tsst tool can be configured in several ways to change the way it processes your survey.

Basic Syntax

tsst [-q] [-v] [-n] [-N] [-o] SURVEY #_OF_ATTEMPTS

SST Optional Configurations

The following tsst modifiers are optional, but sometimes useful when your survey has complex logic.

Option Short Name Default Description
-q Qualified only False Use -q to run SST in a way that will attempt to avoid terminate conditions and aim to provide answers that will result in qualified completes.
-v Skip <validate> False

Use -v to ignore <validate> checks.
 

This is considered cheating. The preferred method is to create a good sst.txt file to specify acceptable responses.

-N No clear data False Use -N if you do not want to clear the existing data in the survey.
-o Respect optional False Use -o if you want a 20% chance that an optional question is skipped. By default, all questions are answered regardless of optional="0" or atleast="#" attributes.
Advanced Options
--quota Divide quotas 1,1 Use --quota 5 to divide quota limits by 5 when running SST. Use --quota 7,4 to divide by 7 to a minimum of 4 rather than 1. For example, attempting 11 qualified SST runs with a quota limit of 100 with --quota 10 specified, the last attempt will OQ.
--quota-increase=# Multiply quotas 1 Use --quota-increase=10 to multiply all quota limits by 10. For example, attempting 100 qualified SST runs with a quota limit of 10 will result in 100 qualified completes instead of 90 overquotas.
--retry=# Try again 5 Use --retry=8 to attempt an SST run 8 times before terminating. This will submit the page 8 times with different data if the survey is terminating. The maximum number of retry attempts is 10 and you can set --retry=0 to disable.
--style Apply styles 0

Use --style=1 to force evaluation of custom style overrides in the project's nstyles file.

Custom styles are ignored by default.

--profile Profiling False

Use --profile to help narrow down which questions are taking a long time to execute. This is often due to complex logic.

Developers only: use --cprof=qlabel to run profiler when executing that question.

-f# Parallel processing 4

Use --f1 to run SST attempts one-at-a-time.

This feature is rarely necessary. You should not increase value beyond 4.

Examples

tsst -qv . 100 will run 100 SST attempts without terminating (hopefully) and ignoring <validate> elements

tsst -qvoNn . 500 will run 500 SST attempts without terminating (hopefully), ignoring <validate> elements, attempting to skip optional questions, without clearing current data and ignoring question styles

3:  Specifying Answers

Use a file named sst.txt in your project's directory to manually declare values for questions and variables in your survey.

e.g. /selfserve/9d3/proj1234/sst.txt

If your SST attempts result in more terminates than qualified completes, then the sst.txt file can help provide valid data for those questions with complex logic.

For example:

<number label="Q1" size="3" verify="range(1, 125)">
  <title>Please enter your age below.</title>
</number>
<suspend/>

<term cond="not Q1.check('18-64')">Q1: Age less than 18 or greater than 64</term>

In the code above, any value that is not within the range 18 - 64 will result in termination. To avoid wasting SST attempts on terminates, we can manually specify the value(s) to use at question Q1 in the file sst.txt:

Q1: 18 24 34 44 54 64

Now SST will only use the values provided and supply answers that will avoid the termination logic altogether.

Here are some other examples:

list: 1 2 3     # specify values for extraVariables (e.g. list)
co: us fr br    # specify values for extraVariables (e.g. co)
Q1: c1 c2       # specify columns to select for a radio question
Q2: r1 r2 r3    # specify rows to select for a radio question
Q3.r1: c1 c2 c3 # specify columns to select for a specific row in a radio question
Q4.c2: r1 r2    # specify rows to select for a specific column in a radio question
Q5.r1: ch1 ch2  # specify choices to select for a specific row in a select question
Q6.c1: 100%     # specify which column to select in a checkbox question
Q7.r1: 50%      # r1 at Q6 will be checked 50% of the time
Q7.r2.c1: 100%  # c1 will be checked for r2 at Q6 100% of the time
Q8: 5 10 20-50  # specify numbers or number ranges for number questions
timeSpent: 50-200 # override the default 30s per question returned

For more information, check out Specifying Answers for Simulated Data (SST).

3.1:  Skipping an Element

You can specify to skip any element with the SST attribute. This prevents the survey stress tester from testing that element. When sst="0" is set, the SST system ignores the element.

For example:

<number label="Q1" size="3" verify="range(1, 125)" sst="0">
    <title>Please enter your age below.</title>
</number>

Additional example:

<term cond="not Q1.check('18-64')" sst="0">Q1: Age less than 18 or greater than 64</term>

For more information, check out Element Attributes.

4:  Additional Components

When SST is ran, the following components of your project are affected:

4.1:  Test Data History & Report

A full report of all SST attempts is stored at /admin/sst/list. You can access your project's test data report using the following link (update the highlighted part with your project):

https://v2.decipherinc.com/admin/sst/list?survey=selfserve/9d3/proj1234

The test data report includes:

  • a list of all SST sessions initiated
  • the number of SST attempts per session
  • the status of each attempt (finished, fatal, fail, loop)
  • fatal error message
  • survey view of the pages seen
  • question coverage
  • page time and traffic load

For more information about the test data report, see Simulated Data History Report.

4.2:  Coverage Report

The Coverage Report shows how many questions have been seen. You can access your project's coverage report in the data downloads menu beneath the Direct to Browser column by selecting Coverage.

You should use this tool to verify that there are no errors in logic. If there are questions in your survey that have been seen few or 0 times, this may indicate an error.

To account for complex logic, you many need to configure the project's sst.txt file to simulate the correct answers in order to traverse those complex paths. Learn more.

4.3:  Progress Bar

The progress bar relies on simulated data. In order for the percentage to line up correctly, it's suggested that you run at least 200 SST attempts through the final version of your survey. This helps establish all of the possible paths that a respondent can take.

If your survey is especially complex, consider running more SST to account for those complex paths.

  • War dieser Artikel hilfreich?