Apklausų tyrimų problemos: metodologija ir kokybė

Tarptautinis specializuotas duomenų analizės seminaras


2011 m. gegužės 23-24 d., KTU
Autoriai: Oliver Lipps (Šveicarijos socialinių mokslų tyrimų fondas, FORS), Brita Dorer (Leibnico socialinių mokslų institutas -GESIS)

 Įvadas  Oliver Lipps pranešimas Brita Dorer pranešimas

Autorius Brita Dorer, GESIS (Mannheim, Germany)
“Questionnaire Translation: the example of the European Social Survey (ESS)”


Seminaro medžiaga

1. Translation method applied in the ESS: TRAPD method
2. Methods for improving translation quality in ESS round 5:
    2.1 Advance Translation
    2.2 Translation Verification by cApStAn
    2.3 Shared Languages
3. PART B – WORKSHOP: Translation Exercises
4. PART B – WORKSHOP: Questionnaire Translation: the example of the European  Social Survey (ESS) - typical translation pitfalls


My intervention on “Questionnaire Translation: the example of the European Social Survey (ESS)” was divided into two parts: The first part was a presentation on, first, the TRAPD method, i.e. the translation method applied in the ESS, and then on several methods applied in the 5th round of the European Social Survey for improving translation quality:

a) Advance Translation;
b) Translation Verification by cApStAn;
c) the Shared Languages approach.

The second was a practical part with exercises to be done by the participants in the classroom and a workshop/discussion character.

1. Translation method applied in the ESS: TRAPD method

The TRAPD method is the standard translation method applied in the ESS. This is an acronym composed of the different steps involved: Translation, Review, Adjudication, Pre-testing, Documentation. A core element is a team approach consisting of people deploying different functions: translators, reviewers and adjudicators.

Translation quality in cross-cultural surveys is crucial because it is essential in order to “ask the same question” in all participating countries. Only by doing so it is possible to achieve data comparability between the participating countries.


  2. Methods for improving translation quality in ESS round 5:
          2.1  Advance Translation

In cross-cultural surveys, one potential source of measurement error arises from questionnaire drafting or from translation.

To reduce the risk of errors stemming from intercultural problems or poor translations, an additional step was introduced for the first time in the fifth round of the European Social Survey (ESS), already during the drafting stage of the source questionnaire:
A so called “Advance Translation” procedure was carried out in two participating countries using two different languages (Swiss-French and Polish). The two Advance Translation teams were asked to perform a problem-oriented translation – following the recommended ESS translation process of parallel translation and team discussion. In both cases, the National Coordinators acted as the reviewer/adjudicator. The purpose of this advance translation was to get input from different cultural and linguistic backgrounds before finalising the source questionnaire for cross-cultural implementation. The two teams worked on the ESS5 Pilot Questionnaire, more or less in parallel with the fielding of the Pilot study. A split approach was used, i.e. both countries translated only parts of the questionnaire. Items were selected which were thought to be of particular interest in the view of finalising the source questionnaire: These were almost exclusively new items from the rotating modules or items that had been modified from former ESS Rounds – and which could therefore still be modified in the English source text.

The quality of the final translations chosen by the two national teams was less important than the comments they provided. The teams were asked to comment above all on translation-related problems, reaching from linguistic or grammar issues to e.g. wording, meaning or intercultural aspects. For each item, the teams had to select from a pre-determined list of problem categories and then to comment on their problems in their own words (in English in order to be understood by all project participants).

The Advance Translation comments were first analysed by the ESS Translation Expert and then forwarded to the Questionnaire Design Teams who considered them when finalising the ESS Round 5 source questionnaire. The advance translation comments led to a number of changes in the final source questionnaire: In many instances, annotations, i.e. footnotes, were added in order to provide additional information about certain words or expressions – and thus trying to avoid inconsistencies between the final translated national versions. Often the wording of source items was amended in a way to facilitate translation into 30+ language versions.

My presentation described the process and highlighted some graphic examples.


2. Methods for improving translation quality in ESS round 5:
  2.2 Translation Verification by cApStAn





Translation verification was introduced in ESS5 as an additional – external – translation quality check before finalising the questionnaire translations. cApStAn is an external provider, specialising in developing linguistic quality assurance and linguistic quality control systems for use in multilingual, multinational and multicultural surveys. It has been involved in many other important international surveys since 2000, amongst others PIRLS, TIMSS, PISA, PIAAC and SHARE. The purpose of translation verification was to improve the overall ESS translation process.
My presentation briefly described the implementation of translation verification in the 5th round of the ESS; in addition, I gave some first conclusions of this exercise.

2. Methods for improving translation quality in ESS round 5:
  2.3  Shared Languages




In its 5th round, the ESS has reinforced its focus on translation into its so-called ‘shared languages’: these are languages fielded in more than 1 country, as e.g. French which is used in France, Belgium and Switzerland. Although it has been recommended that the national teams concerned reconcile their versions before going into the field, this is done in different manners. The focus in ESS5 was to at least document the different approaches better than previously.

  3. PART B – WORKSHOP: Translation Exercises

I started the second, practical, part with some translation exercises in order to allow all participants to experience by themselves what problems may arise when translating an English master questionnaire into their mother languages:

Workshop “Questionnaire Translation”

Translation Exercises
[Taken from the Final ESS Round 2 Questionnaire]

Text Box: Tasks:  1.	Translate individually:	- What problems do you encounter?   				- Do you have comments?    2.	Translate in a team including team discussion:  				- What differences do you feel?  				- Which way do you prefer?     

Footnotes: 46=44



Translation and adaptation (Mainly items B12 and B16)


  4. PART B – WORKSHOP:uestionnaire Translation: the example of the European
         Social Survey (ESS) - typical translation pitfalls

In the second half of my workshop part, I presented some typical translation pitfalls known from the ESS translation history. I discussed several of these issues with the audience which seemed particularly problematic to (some of) the participants. 













Behr, D. (2009). ‘Translationswissenschaft und international vergleichende Umfrageforschung: Qualitätssicherung bei Fragebogenübersetzungen als Gegenstand einer Prozessanalyse’. Bonn: GESIS. [Translation Research and Cross-National Survey Research: Quality Assurance in Questionnaire Translation from the Perspective of Translation Process Research]

Braun, M. and Harkness, J. (2005). ‘Text and Context: Challenges to Comparability in Survey Questions’ in: Hoffmeyer-Zlotnik, J.H.P., Harkness, J. (Eds.). Methodological Aspects in Cross-National Research. ZUMA-Nachrichten Spezial No. 11. Mannheim: Zentrum für Umfragen, Methoden und Analysen.

Brislin, R.W. (1986). ‘The Wording and Translation of Research Instruments’, in: Lonner, W.J., Berry, J.W. (ed.). Field Methods in Cross-Cultural Research. Beverly Hills: Sage.

Dept, S., Ferrari, A. and Wäyrynen, L. (2008). ‘Comparative Overview of Test Adaptation and Translation Verification Procedures in Three International Surveys’. Paper presented at the international conference on survey methods in multinational, multiregional, and multicultural contexts (3MC), Berlin, Germany. Retrieved April 16, from 3mc2008_proceedings/session_09/Dept.pdf.

Dept, S., Ferrari, A. and Wäyrynen, L. (2010). ‘Developments in Translation Verification Procedures in Three Multilingual Assessments: A Plea for an Integrated Translation and Adaptation Monitoring Tool’ in: Harkness, J., Braun, M. et al. (Eds.). Survey Methods in Multinational, Multiregional, and Multicultural Contexts. New York: John Wiley and Sons.

European Social Survey (2009). ‘Parallel questionnaire design in the ESS: Advance translation as a tool for promoting cross-cultural input’. Mannheim, European Social Survey GESIS.

European Social Survey (2010). ‘ESS Round 5 Translation Guidelines’. Mannheim, European Social Survey GESIS.

Göpferich, S. (2008). ‘Textproduktion im Zeitalter der Globalisierung’. Tübingen: Stauffenburg Verlag.

Harkness, J. (1995). ‘ISSP Methodology Translation Work Group Report 1995, Report to the ISSP General Assembly at the 1995 Cologne ISSP meeting’.

Harkness, J. and Schoua-Glusberg, A. (1998). ‘Questionnaires in Translation’ in: Harkness, J. (Ed.). Cross-Cultural Survey Equivalence. ZUMA-Nachrichten Spezial No. 3. Mannheim: Zentrum für Umfragen, Methoden und Analysen.

Harkness, J., Langfeldt, B. and Scholz, E. (2000). ‘ISSP Study Monitoring 1996-1998. Reports to the ISSP General Assembly on monitoring work undertaken for the ISSP by ZUMA, Germany’. Mannheim.

Harkness, J. (2003). ‘Questionnaire Translation’ in: Harkness, J., van de Vijver, F. and Mohler, P. (Eds). Cross-Cultural Survey Methods. New York: John Wiley and Sons.

Harkness, J., Pennell, B., and Schoua-Glusberg, A. (2004). ‘Questionnaire Translation and Assessment’ in: Presser, S., Rothgeb, J., Couper, M., Lessler, J., Martin, E. and Singer, E. (Eds.). Methods for Testing and Evaluating Survey Questionnaires. New Jersey: John Wiley and Sons.

Harkness, J. (2005). ‘ISSP Translation Work group Report to the ISSP General Assembly’. Mexico City.

Harkness, J. (2007). ‘Improving the comparability of translations’ in: Jowell R. et al. (2007). Measuring Attitudes Cross-Nationally. Lessons from the European Social Survey. London: SAGE Publications.

OECD (2009). ‘PISA 2006 Technical Report’. Paris.

van Nes, F. et al. (2010). ‘Language differences in qualitative research: is meaning lost in translation?’ in: European Journal of Ageing, 7(4):313-316. Springer.


Nr.1  2009 07-11
Nr.2  2009 12-2010 02
Nr.3  2010 03-05
Nr.4  2010 06-08
Nr.5  2010 09-11
Nr.6  2010 12-2011 02
Nr.7 2011 03-05
Nr.8 2011 06-08
© KTU Politikos ir viešojo administravimo institutas
Atnaujinta 2015-06-18