Author: GRC Practice Director, Alex Hollis.
Blog Series Introduction
In this Third Party Risk Management blog series, Alex Hollis will guide you through developing effective information gathering for third parties using five key steps to the formulation of a third party questionnaire. The webinar is available on-demand via BrightTALK here.
There are five key steps to the formulation of a third party questionnaire:
- Requirements – establishing the needs of the organization both in terms of the risks that need to be managed and the compliance needs from regulation and any stakeholder commitments.
- Research – obtaining an understanding of the types of information needed to satisfy the requirements and prioritizing the needs among the various types of third parties the organization has.
- Planning – consideration for the method, structure, and number of assessments (this can also include non-questionnaire approaches such as audits and interviews)
- Writing questions – Formulating the actual questions themselves and the method of response.
- Testing – Obtaining validation and identifying any areas of improvement.
In the tenth installment, we will continue to discuss the importance of shortening your questionnaires to ensure you have your readers attention. Alex will explain how simply providing the most suitable questionnaire options, you can increase the reliability of your respondents’ answers. The blog includes real-life examples.
6) Don’t use complex layouts/grids
When you know the subject of a set of questions well it can be tempting to use a compressed format such as a table/gird to ask and answer multiple common questions around a domain.
This may be how the person receiving the data wants to see it as they have time to become familiar with this design, but the respondent does not have the same familiarity. Also, remember that the respondent will be working with some other assessments using different layouts.
Asking questions one at a time with no gird is easier for respondents to answer. The only exception to this would be if you are working with a very standard approach that the respondent will be familiar will completing.
7) Adding labels and placeholders
When confronted with a blank line or empty box, a respondent will need to evaluate what type of response is needed. They will also feel slight inertia in committing to putting data into the empty box.
Placeholder text or labels provide an immediate reassurance as to what needs to go into the box, further defining what the respondent is expected to do and also reduces that inertia around populating the empty box.
8) Make Questions Unbiased
To collect data that is reliable, we should ensure that our questionnaire does not lead or bias the respondent in any way.
Given the relationship between vendor and client, there is already pressure on the respondent which creates a bias towards potentially false-positive answers. We should, therefore, be careful to ensure that we are doing the following.
There is sometimes the temptation to leverage the third party questionnaire to push third parties to achieve a far higher standard than the organization themselves achieve.
In terms of risk, it doesn’t matter whether it is internal or through a third party. As such a similar approach to risk management should prevail. The accuracy of measurement and identification of risk is what is important.
Ensure that questions are asked openly and with the intention of discovering the information needed to make the required decisions. The goal should never be to influence or drive a behavior change.
Make clear to the respondent that positive or negative answers are equally acceptable. Clearly, there may be a level of risk which cannot be sustained, but the importance of identification and positive remediation is more important.
If we are not careful, respondents may acquiescence and offer answers which are more agreeable or positive to questions.
9) Provide all options, don’t overlap
When providing a pre-defined choice to the user, all possible options must be accommodated.
| Please select the data subjects groups for the personal data you process | Customer Employee |
The above question is asking about data processing around personal information. The list only allows us to select Customer or Employee. What about marketing tools which process data around website visitors, who might not be customers?
This finite list is too restrictive; it doesn’t provide all the possible options. It also doesn’t provide an ‘Other’ option to collect the data.
The respondent is now open to interpret that the organization only cares about these two data subject groups or may use an extended definition of the customer. In either case, we are introducing inaccuracies in the data.
The second point is not to allow options to overlap. Take the above question again with some updates.
| Please select the data subjects groups for the personal data you process | Website Visitors Prospective Customers Customer Employee Other: ________ |
Website visitors may be prospective customers, customers and employees. As such should the respondent be selecting all options? What if they only select Website visitors and Prospective customers but don’t select the others?
Again there is no clarity around the answers and that allows for interpretation, which in turn introduces inaccuracy.
10) Make sure numeric ranges are only as broad as is necessary
Numeric ranges provide a lower commitment from the respondent, but you must ensure the intervals between the ranges are appropriate for all respondents. The ranges will have an influence on the respondents.
Rockwood, Sangster and Dillman (1997) asked students how many hours they studied. One group selected from a set of options in a low range and the other within a high range. 23% of students in the low range reported 2.5 hours, whereas 69% of students in the high range reported 2.5 hours. The choices clearly influenced the responses.
Researchers think that respondents calibrate their answers to appear more equal to their peers. Respondents don’t like to appear on either of the extremes.
When it comes to third-party assessments, there is also a desire to seem less risky as such respondents will push towards the more beneficial side of the scale.
| Estimated number of records held | <1,000 1,000 – 99,999 100,000 – 1,000,000 >1,000,000 |
If the number of records held is 100,500, there may be a temptation to push down into the bracket below by using the estimation to account for the error depending on the perceived risk that might dramatically change the opinion from high to low.
Without introducing too many options gradually increasing the scale won’t allow respondents to move too far through the estimation error.
| Estimated number of records held | <1,000 1,000 – 9,999 10,000 – 99,999 100,000 – 1,000,000 >1,000,000 |
In the long run, provided the organization has the data, you might be better off sticking with the numeric response and then banding the result for yourself.
11) Consider using forced choice instead of “check all that apply.”
Select all that apply asks the respondent to look through a set of choices and positively confirm the answer.
Check all that apply:
| Please select the data subjects groups for the personal data you process | Customer Employee |
Forced Choice:
| Please select the data subjects groups for the personal data you process | Customer | Yes/No |
| Employee | Yes/No |
Research into the two styles of approach report that a forced choice provides more accurate results from respondents as they are required to affirm the choice rather than provide an implied No.
Respondents to a check all that apply are scanning the list to find things that apply rather than actively considering each option.
The downside is that forced choice does require more cognitive effort, a short list of potential answers to a question which doesn’t require too much recall may be better in this style.
Next Week…
Stay tuned for the next blog in this series TPRM Blog 11- How to Write Effective Open Questions. The blog will discuss how to get the most detailed and honest answer from an open question.
To view the previous blogs in the series click here.
See you next week!
A