TPRM Blog 6-Writing Clear Questions

Third Party Risk Management Blog Series Introduction

In this Third Party Risk Management blog series, GRC (Governance Risk & Compliance) Practice Director, Alex Hollis will guide you through developing effective information gathering for third parties using five key steps to the formulation of a third party questionnaire. 

There are five key steps to the formulation of a third party questionnaire:

• Requirements – establishing the needs of the third party organisation both in terms of the risks that need to be managed and the compliance needs from regulation and any stakeholder commitments.
• Research – obtaining an understanding of the types of third party information needed to satisfy the requirements and prioritising the needs among the various types of third parties the organisation has.
• Planning – consideration for the method, structure, and number of third party assessments (this can also include non-questionnaire approaches such as audits and interviews)
• Writing questions – Formulating the actual third party questions themselves and the method of response.
• Testing – Obtaining validation and identifying any areas of improvement.

In the sixth installment, Alex will be exploring the importance of clear communication for collecting accurate information from your third parties. He will be providing 8 key rules for how to write well thought out questions, three of which are exclusive to this series.

Writing the Questions

When drafting the questions, there are some simple techniques to improve the collection of information and help manage third party risk and the assessment fatigue experienced by the respondent.

Being Clear

Communication is about getting the thoughts and concepts from your own mind into someone else, despite how often we communicate we all still make mistakes from time to time. Even face-to-face with the full benefit of voice, tone, inflexion, facial and gestural expressions, knowing your audience as well as the ability to ask clarifying questions to get the full context, we all still on occasion miss the true meaning of what another person was trying to convey.

Questionnaires remove many of the cues that we would have during a face to face interaction and, also often must make general assumptions about the audience without the ability to modify or rephrase the question for a specific respondent (without introducing huge inefficiency).
Here are some rules to help make sure questions are clear.

1) Unit of measurement – when specifying questions be clear on what you are expecting a third party respondent to give you.

This question is asking when the last time the software was actively tested to find exploitable vulnerabilities.

There is no clue to the unit of measurement; it could be the specific date, the month, the year. It could also be expressed as the number of days, weeks, months, years since the last test.
Depending on the way this will be displayed this to the respondent there may be options to format the input as a date, use placeholder text or suffixes to make clear the expectation.

2) Use the vocabulary of the respondents – Learn and use the vocabulary of your respondents.
This is often why when designing questions it is useful to work closely with a set of the target respondents either by interviewing or in a focus group.

Avoid using technical jargon from your own organisation and be careful about assuming technical understanding from a recipient. Compliance questions should not seek to test the companies’ comprehension of the domain. Additionally, don’t make assumptions about the capabilities of the organisation. Unfortunately, a misunderstanding here might result in a false positive response, meaning there is a third party risk that has not been uncovered.

3) Using clear words and phrases – Even with the right vocabulary, you can still confuse a respondent.

This question uses the ambiguous term ‘who handles’, do you want the accountable person, the responsible person, the agency who likely does the checks. It’s unclear and open to interpretation.

The customer can be interpreted in a lot of ways; some might have a subscription, others might provide units which can be purchased over and over by a single customer which then brings in the time frame, how many today, last month, last year.

4) Be clear about the subject of your question – The subject could be the individual, the department, the company, the product or the service. You must be clear to whom the question is directed.

An individual could interpret this as a question directed to them. After all we likely have an attestation which says “Have you answered all questions to the best or your knowledge?”.
It could also be interpreted as a question about the support team or supporting services. It could be a general question about the company’s use of notifications or a product capability question.

Put simply just because it seems obvious to you, doesn’t automatically mean that everyone else which have the same perspective.

For midsized organisations who don’t own their own offices then physical security controls may be provided as part of the lease. The same goes to data centres; few organisations own the physical security around a data centre. As such the company indirectly has CCTV, it is benefiting from the control, but it does not own or manage this control.

“You” is a word that has to be treated very carefully to ensure it is known who is meant.

5) Ask one question – Make sure you are only asking one question at a time.

Here a respondent who has an encryption policy but nothing around destruction might misunderstand thinking the polices are covering a common area or deliberately answer positively with a clarifying comment.

It doesn’t matter whether you use “and” or “or” in either case you are combining to questions and inviting ambiguity on which question is being answered.

Exclusive to this third party risk management blog series, 3 additional rules:

6) When asking about percentages, be clear about the base.

Although the timeframe is clear, it is open to interpretation around whether or not business hours are included; planned maintenance is often left out. This can lead to incorrect comparisons between service providers.

An additional consideration for this is that by asking the respondent to calculate the percentage, you are asking them to carry out a form of processing on the third party data. This leaves it open to manipulation but also incorrectly calculating.

We could instead ask:

This gives us the data from which we can create a percentage if one is needed.

In the last example, the denominator is known, but sometimes you might need to make the denominator and the numerator dynamic. In which case you can do this:

Sometimes you wish to arrive at a percentage based on a number of previous iterations. Often using a base of 10 or 100 is useful.

7) Make sure the questions and answers match.

It appears too obvious to be worth stating, but it often happens when updating wording.

This disconnect between the question being asked and the answer here is a very obvious one. However, the disconnect can be more subtle.

Here the respondent is being required to convert the level to whether they think security is or isn’t important, and then convert that onto a scale of agreement or disagreement.

When the question and answers choices match, questions are clearer and easier for respondents to answer. Respondents aren’t fatigued by any additional thinking.

8) Use formatting such as bold, underlining, italic, and/or capitalisation to highlight keywords and phrases.

In natural language, we use tone, inflexion and gesticulation to emphasize particular words and phrases. Written language does not have this same luxury when we need to draw attention to something we must use other techniques.

The use of formatting and styling (bold, underlining, italics etc.) can be used to draw attention. This is particularly useful where a respondents attention needs to be drawn to some kind of distinction from other questions in a series or context change.