Author(s):
Ana Paula Teixeira de Almeida Vieira Monteiro1 and Alexandre
Bastos Fernandes2
Background
The role of migration in European population change is being
debated during recent years as a result of growing concerns about issues such
as demographic ageing, shortages of working age populations and payment of
pensions among others [1]. In 2015, Europe is suffering from the worst refugee
crises since World War II. These refugees are fleeing war, violence and
persecution in their country of origin.
Immigrants and asylum-seekers have a higher risk of mental
illness due to the fact that they often have been exposed to extreme
conditions--for instance, forced migration, significant personal losses and
other human rights violations-simply due to the fact that they are refugees in
a foreign country, with different cultural norms [2].
Mental health issues including major depression disorder,
post-traumatic stress disorder, and general anxiety disorder are common among
newly arrived immigrants and refugees. While many immigrants and refugees are
resilient, traumatic experiences and migration stressors have a great impact on
their mental well-being [3-5].
As the largest group of health professionals in the world,
nurses are the key to providing cost effective mental health care to this
vulnerable populations, meeting the immigrants and refugees' complex mental
health needs. Nurses are key to healthcare for refugees and migrants [6].
However, most nurses lack competence and confidence in dealing with the
distress and mental health problems of refugees and migrants. Lack of cultural
competence in mental health among nurses limits the potential to provide
high-quality care for the growing number of people with diverse backgrounds
[7]. An additional factor that can be a conundrum when examining standards for
culturally competent mental health care is the lack of uniformity in educating mental
health providers about cultural competence and inconsistencies in how cultural
competence is defined [8].
Cultural competence can be seen as a necessary set of skills
for mental health nurses to attain in order to render effective patient-centred
care. Still, there is limited literature available identifying and describing
the instruments that measure mental cultural competence in nursing
professionals. These instruments are crucial to assess their cross-cultural
strengths and weaknesses in order to design specific training activities or
interventions that promote multicultural competence in mental health care [9].
Portuguese is the world's sixth most spoken language (in
America, Africa and Europe there are over 230 million Portuguese speakers) and
millions of nurses are using Portuguese as the first language in health care.
In spite of this fact, there are no Portuguese validated measurement
instruments to assess cultural competence in mental health among nurses.
Cultural competency has been defined as a set of congruent
behaviors, attitudes and policies that come together in a system, agency or
among professionals that enable that system, agency or those professions to
work effectively in cross-cultural situations. It has been seen as the ongoing
process in which the health care provider continuously strives to work
effectively within the cultural context of the client [10]. Thus, cultural
competence is active, developmental, and is aspirational rather than achieved
[11].
The tripartite cultural competence model [12, 13] has three
components: 1- Cultural awareness, 2-Cultural knowledge, and
3-Cultural skills . First, the awareness component
refers to the mental health professional's awareness of one's own worldview and
cultural biases. Cultural awareness is the self-examination
and in-depth exploration of one's own cultural and professional background. In
other words, this process involves the recognition of one's biases, prejudices,
and assumptions about individuals who are different. A culturally competent
professional is one who is actively in the process of becoming aware of his or
her own assumptions about human behavior, values, biases, preconceived notions,
and personal limitations. Second, cultural knowledge is the
process of seeking and obtaining a sound educational foundation about diverse
cultural and ethnic groups. In order to obtaining this knowledge base, the
health care provider must focus on the integration of three specific issues:
health-related beliefs and cultural values, disease incidence and prevalence,
and treatment efficacy [14]. A Multicultural knowledge requires
mental health givers to be knowledgeable about various cultural factors that
might influence the therapeutic process. A culturally competent professional is
one who actively attempts to understand the worldview of culturally diverse
populations--values, assumptions, practices, communication styles, group norms,
biases, and personal experiences.
Multicultural competence of mental health practitioners
includes the specific skill of adapting treatments and practices to better
match the particular needs of the client [15].
The ethnic composition Portuguese population is becoming
increasingly diverse in the last two decades. The Portuguese mental health care
system is struggling to address some of the specific challenges presented by
recent immigration from Africa, Brazil, Eastern Europe and China [16]. For
mental health care providers and specifically nurses, the need to provide
culturally appropriate and competent care is recognized as essential. Still,
little attention has been given to the development of multicultural
competencies among Portuguese health care professionals in mental health
settings [17,18]. A study with 22 Portuguese nurses, aiming to analyse personal
experiences and significant situations of nursing care in multicultural
contexts, highlights key multicultural issues that are important for mental
health and psychiatric nursing care: therapeutic environment management, roles,
speech patterns, and therapeutic communication with minority clients and
families [19]. Currently, the instruments to assess cultural competence in
nurses and nursing students are self-administered and based on individuals' own
perceptions. The instruments are commonly utilized to test the effectiveness of
educational programs designed to increase cultural competence [20]. Therefore,
instruments that accurately and reliably measure the knowledge, attitudes, and
skills reflecting cultural competence in Portuguese mental health nurses are
scarce.
A study was developed to evaluate the cultural diversity
competence of Portuguese child care workers (N = 51) [21]. The variables
integrated a self-report measure- a cultural competence questionnaire, and an
objective measure for evaluation of cultural diversity competencies. But this
self-report measure had some limitations, namely the small size of the sample
in the validation study, the low Alpha Cronbach's in two subscales and the fact
that it was targeted to child care workers.
Methods
Aim
The aim of this study was to translate and test the
psychometric properties of the Portuguese version of the Multicultural Mental
Health Awareness Scale--MMHAS.
Design
A psychometric research design was used with content and
construct validity and reliability. Reliability was determined with Cronbach's
alpha coefficient.
Sample/participants
A convenience sample of Registered Nurses (RN) was recruited
online using the LimeService/ LimeSurvey hosting platform. The
research questionnaire was made available online, and participants were invited
to participate by email, social networks, and trough professional contacts. The
survey took approximately 20 to 30 min to complete and was accessed with a link
toLimeService/ LimeSurvey hosting platform. The sample is composed
of 306 subjects - aged between 21 and 68 years (with an average age of 35.4
years and a standard deviation of 9.8 years); 25.2 % female and 74.8 % male,
who agreed to participate in the study and matched the following inclusion
criteria: to be a RN in Portugal and proficiency in Portuguese Language.
Data collection
Data were collected using the Portuguese version of
the Multicultural Mental Health Awareness Scale -MMHAS, along
with a socio-demographic questionnaire.
Multicultural mental health awareness scale-MMHAS
The original Multicultural Mental Health Awareness Scale
(MMHAS) was designed to provide a psychometrically sound instrument to
successfully assess professional multicultural competence in mental health and
the effectiveness of a multicultural training program in mental health. Items
on the new scale were generated in order parallel the Queensland Transcultural
Mental Health Centre (QTMHC) training program's objectives. Results indicated a
scale with 35 items and three factors with moderate to high factor loadings and
satisfactory psychometric properties. The three factors found to be
interrelated were: Multicultural Counselling Awareness, Multicultural
Counselling Knowledge, and Multicultural Counselling Skills, in line
with the multicultural counselling competencies. The Cronbach's alpha for the
total scale was .91. The internal consistency of theMulticultural
Counselling Awareness subscales was .89, .92 for the Multicultural
Counselling Knowledge subscale, and .90 for the Skills subscale.
The MMHAS is unique as it not only reflects language issues and correct
intervention strategies, but also assesses the ability to address service
barriers and to work with interpreters [22].
Procedures
The questionnaire was translated using the back translation
system by two bilingual translators. After the translations process, a
committee including one expert in mental health, one expert in English Language
and Literature, also bilingual and, the researcher discussed discrepancies and
agreed upon an integrated version of the translation. A group of nursing
students reviewed the questionnaire with the investigator to check the
appropriateness of the translations and resolved remaining discrepancies in
translations. Consensus in terms of semantic, idiomatic, experiential, and
conceptual equivalence was reached and a final version of the MMHAS Portuguese
version was provided. The study sample was identified using a convenience
sampling method. After a process of translation and transcultural validation,
the Portuguese version of MMHAS, along with a socio-demographic questionnaire,
were applied to a sample of 306 Portuguese nurses. The research questionnaire
was made available online, and participants were invited to participate by
email, social networks, and trough professional contacts. The information of
the study was disseminated online in several nursing professional networks and
social network sites (Facebook, Portuguese Nursing Blogs; Portuguese Nursing
sites). Only a few participants were directed notified, for ex. with a
personnel email with a link.
The survey took approximately 20 to 30 min to complete and
was accessed with a link toLimeService/ LimeSurvey hosting
platform. LimeSurvey provides a web-based product for internet
surveys with more than twenty different question formats, in addition to survey
tokens that prevent repeat participants, and branching surveys that offer
different questions depending on prior responses. Moreover, the survey
administrator can control how the survey interface will appear to participants
through a built-in editor. Surveys were available online from November 2013 to
February 2014. Once the recruitment period had ended all data was condensed
into a file, downloaded into a database, exported into SPSS and, statistical
analysis was conducted. It was not possible to determinate a response rate.
Ethical considerations
Internet data collection can raise particular, sometimes
non-obvious challenges in adhering to ethical principles. In this study we
followed the Ethics Guidelines for Internet Mediated Research [23] and the
American Psychological Association's Guidelines [24]. Study inclusion required
informed consent. Participants were informed that they could withdraw from the
study at any time. Regarding ethical aspects in research, it is worth
highlighting that authorization to make cultural adaptation and validation of
the MMHA for the Portuguese reality was obtained from the author of the
original instrument. The study was formally approved by the Ethical Committee
for Health Research of Coimbra Nursing School.
Data analysis
Data analyses were completed with the use of Statistical
Package for the Social Sciences version 20.0 (SPSS, Chicago, IL, USA). To
explore the MMHAS internal consistency, Cronbach's alpha was
obtained, as well as item-total correlations and alpha values when the item was
deleted. Exploratory factor analysis (principal component analysis) was carried
out to identify factor structure of MMHAS (Portuguese version).
Results
Participants
The sample was composed of 306 subjects--aged between 21 and
68 years, with an average age of 35.43 years and a standard deviation of 9.85
years; 299 males (74.8 %) and 77 females (25.2 %).
Demographic
characteristics of the respondents are summarized in Table 1.
Table
1: Sociodemographic characterization of the sample [see PDF
for image]
Principal component analysis
In our exploratory factor analysis, the Kaiser Mayer Olkin
Measure and the Bartlett's chi square tests were checked for the
appropriateness of data for factor analysis. Both the adequacy of the sample
and the use of factor analysis on the data were confirmed.
The Kaiser-Meyer-Olkin (KMO) measure of
sampling adequacy was 0.949, exceeding the benchmark value of 0.60. The Bartlett's
Test of Spehericity ([chl] 2 = 7776.696; p = 0.000) was
statistically significant, supporting the factorability of the correlation
matrix. A principal component analysis (PCA) with orthogonal varimax
rotation and combined scree plot test and parallel analysis yielded a
three factor solution that accounted for 59.4 % of the total variance explained
(Table 2). A principal component analysis (PCA) with orthogonal varimax
procedure was employed to rotate the factors to a simple structure in order to
determine the number of factors to retain. Items with a loading of greater than
0.50 were retained for a specific factor. Split items were retained to factors
if the square of the loadings for a factor was >50 % that of its loading on
any other factor (Table 2).
Table
2: Loadings (item-component correlations) obtained by PCA [see
PDF for image]
The
items of the MMHAS Portuguese version total scale can be divided into three
subscales:
*
Factor 1-Cultural Awareness , with thirteen items (12, 13, 14, 15,
16, 17, 18, 19, 20, 21, 22, 23 and 24).
*
Factor 2-Cultural Knowledge , with eleven items (1, 2, 3, 4, 5, 6,
7, 8, 9, 10 and 11).
*
Factor 3-Cultural Competence/Skills , with eleven items (25, 26,
27, 28, 29, 30, 31, 32, 33, 34 and 35).
As
can be seen from Table 2, all the items (except item 1 and item 2) load onto
their original subscales with loading values of .90 or greater in each case.
The Item 1-"My knowledge of various cultures is" in the
original questionnaire, and in the Portuguese version "o
meu conhecimento sobre outras culturas ñ "-in the original study of
the MMHA loaded in the Factor 1 (Cultural Awareness) and in the Portuguese
version loaded in the Factor 2 (Cultural Knowledge). Also the Item 2-"My
knowledge of acculturation is " in the original questionnaire, and
in the Portuguese version "O meu conhecimento sobre o processo de
aculturação ñ "-in the original study of the MMHA loaded in the
Factor 1 (Cultural Awareness) and in the Portuguese version loaded in the
Factor 2 (Cultural Knowledge). The variations can be explained by some
cultural, semantic and linguistic differences between the Portuguese and
English versions. The mean inter-item correlations, which can be regarded as an
indicator of the homogeneity of the scale, were also computed. In the sample,
the mean inter item correlations are rather high.
Internal consistency reliability
The overall reliability coefficient Cronbach's alpha, is
based on the average correlation of items within the scale. Cronbach's alpha
coefficient of reliability of the MMHAS Portuguese version is 0.958. For the
subscale Multicultural Awareness Cronbach's alpha is 0.938;
for the subscaleMulticultural Knowledge Cronbach's alpha is 0.927
and, for the subscale Multicultural Competence/Skills Cronbach's
alpha is 0.922 (Table 3).
Table
3: Reliability Statistics-Cronbach's [alpha] coefficients of
reliability [see PDF for image]
The
extent to which individual items affect the reliability of the scale can be
examined by calculating Cronbach's alpha when each item is removed from the
scale. The removal of any single item would not significantly increase the
reliability of the scale: in all the items of the scale where the alpha if
deleted isn't higher than the overall alpha and in subscales (Table 3).
Discussion
Cultural
competence is an essential component in rendering effective and culturally
responsive services to culturally and ethnically diverse clients [25]. Because
providing culturally competent care is essential in nursing, the measurement of
cultural competence and its effect on patient outcomes is central to the
discipline. Despite the limitations associated with existing instruments, there
is much value in the initial assessment of cultural competence they provide as
well as tracking measurements of cultural competence over time. The aim of the
study was to investigate whether the Portuguese version of the MMHAS has
satisfactory psychometric properties. The procedures of translation and
cultural adaptation represented no major problems and gave rise to a reliable
Portuguese version of the MMHAS. PCA was carried out in order to re-examine the
factor structure of the scale. The three-factor solution accounted for 54.9 %
of the total variance. The items of the MMHAS Portuguese version total scale
can be divided into three subscales: Cultural Awareness ,Cultural
Knowledge , and Cultural Competence/Skills ,
according the tripartite cultural competence model [26]. Also some
transcultural nursing models underlies the cultural awareness, cultural
knowledge, and cultural skills as crucial components in the process of cultural
competence in the delivery of healthcare services.
Excellent
Cronbach's alpha coefficients for the three subscales and
total scale, and satisfactory item-total coefficients for the correspondent
items confirmed that the MMHAS subscales are internally consistent, with the
correspondent items properly correlated with each other [27-29]. The results
for internal consistency were similar to those obtained by in the original study.
In
this study, 74.8 % of the respondents were male, which does not match the
sociodemographic composition of Portuguese professional nurses. Some emerging
literature points out the correlation between demographic characteristics of
subjects and online survey response behavior by investigating how
socio-demographic factors, gender in particular, affect online survey response
behavior. This is because differences in the way females and males inhabit
cyberspace may exaggerate the effects of differences in how females and males
undergo social exchange, resulting in differences in online survey response
rates [30]. Another explanation is that some of the participants were
psychiatric/mental health nurse practitioners, with a considerable number of
male nurses.
The
MMHAS displayed an apropriate internal consistency. The findings show that the
Portuguese version of the MMHAS adequately addressed the original concepts and
dimensions, and demonstrated a high level of equivalence with the original
version. The findings indicate a remarkable consistency in the factor structure
of the Portuguese version. The concepts embedded in this cultural competence
questionnaire, provide an ideal structure for educational programs. The
Portuguese version of the MMHAS can be used to evaluate the effectiveness of
multicultural competency training programs in Portuguese-speaking mental health
nurses. The scale can also be a useful in future studies of multicultural
competencies in Portuguese nurses.
Limitations
This study has some limitations including the sample
representativeness (74.8 % male), which does not match the sociodemographic
composition of Portuguese professional nurses. Another limitation of this study
was the use of a convenience sample, with the inherent bias. Future studies
should have larger sample sizes and include gender representative samples, and
look at the psychometric performance of the MMHA.
Conclusion
The results of this study support the construct validity and
reliability of the Portuguese version of MMHAS, proving that is a reliable and
valid measure of multicultural counselling competencies in mental health. This
study suggests that MMHAS could be an important addition to the compendium of
instruments used to assess Multicultural Mental Health Competences in Nurses,
and also contributes to improve culturally sensitive nursing care.
The MMHAS Portuguese version can be used to evaluate the
effectiveness of multicultural competency training programs in
Portuguese-speaking mental health nurses. The scale can also be useful in
future studies to access multicultural competencies in Portuguese-speaking
nurses all around the world.
Ethics and consent to participate
Study inclusion required informed consent. Participants were
informed that they could withdraw from the study at any time. The study was
formally approved by the Ethical Committee for Health Research of Coimbra
Nursing School-Comissão de Ética da Unidade de Investigação em Ciências
da Saúde: Enfermagem da Escola Superior de Enfermagem de Coimbra ,
Portugal, Committee's reference number:176-072013.
Consent to publish
Not applicable.
Availability of data and materials
All the data supporting these findings is contained within
the manuscript.
Abbreviations: MMHAS:
Multicultural Mental Health Awareness Scale; PCA: Principal components
analysis; QTMHC: Queensland Transcultural Mental Health Centre; RN: Registered
Nurses; SPSS: Statistical Package for the Social Sciences
Competing interests: The
authors declare that they have no competing interests.
Authors' contributions: Ana Paula Monteiro (APM) and Alexandre Bastos (AB) conceived
the study and developed the study material, and conducted the forward and
backward translations of the Multicultural Mental Health Awareness Scale-MMHAS.
AB carried out data collection. AB conceived and participated in the design of
the study and performed the statistical analysis. APM, and AB analyzed the
data, and APM drafted the manuscript. All authors read and approved the final
manuscript.
Acknowledgements: This
study was carried out at the Coimbra Nursing School, Coimbra, Portugal. We
would like to extend our appreciation to Nigar G. Khawaja for granting the
permissions to translate Multicultural Mental Health Awareness Scale-MMHAS and
to use the Multicultural Mental Health Awareness Scale-MMHAS Portuguese
version. The authors are very grateful to all the nurses who accepted to be
part this study.
Funding
No
funding was obtained for this study.
Correspondence:
Ana Paula Teixeira de Almeida Vieira Monteiro:
Author details: 1 Coimbra Nursing School, Rua 5 de Outubro-Apartado 55,
3001-901, Coimbra, Portugal. 2 Centro Hospitalar e
Universitário de Coimbra (CHUC), Praceta Prof. Mota Pinto, 3000-075, Coimbra,
Portugal.
Article history: Received
14 October 2015 Accepted 4 May 2016 Published online 17 May 2016
References
1.
Population
division: replacement migration. In is it a solution to declining and ageing
populations?. New York: UN; 2000.
2.
Health
and Human Rights Info WEBSITE. 2014. Accessed 8 September 2015. Available in
http://www.hhri.org/thematic/asylum_seekers.html.
3.
Pumariega
AJ, Rothe E, Pumariega JB. Mental health needs of immigrants and refugees.
Community Ment Health J. 2005;41(5):581.-597 doi: 10.1007/s10597-005-6363-1.
4.
Monteiro
AP. Migração e saúde mental, (migration and mental health). Viseu: PsicoSoma;
2011.
5.
Almeida
Vieira Monteiro AP, Serra AV. Vulnerability to stress in migratory contexts: a
study with eastern European immigrants residing in Portugal. J Immigr Minor
Health. 2011;13(4):690.-696 doi: 10.1007/s10903-011-9451-z.
6.
International
Council of Nurses-ICN. Press release . 11 September 2015.
Geneva, Switzerland. Accessed 28 Sept 2015. Available in:
http://www.icn.ch/images/stories/documents/news/press_releases/2015_PR_24_Refugee_Migrant_healthcare.pdf.
7.
Lehman
D, Fenza PJ. The Shift towards Cultural Competency in the Nursing Care
of Older Adults.An Orange Paper from Mather LifeWays, 2011. Accessed 1 Jul
2014. Available in:
http://www.matherlifewaysinstituteonaging.com/wp-content/uploads/2012/03/The-Shift-Towards-Cultural-Competency-in-the-Nursing-Care-of-Older-Adults.pdf.
8.
Nardi
D, Waite R, Killian P. Establishing standards for culturally competent mental
health care. J Psychosoc Nurs Ment Health Serv. 2012;50(7):3.-5 doi:
10.3928/02793695-20120608-01.
9.
Geron
SM. Cultural competency: How is it measured? does it make a difference?.
Generations. 2002;26(3):39.-45.
10.
Campinha-Bacote
J. Cultural diversity in nursing education: issues and concerns. J Nurs Educ.
1998;37(1):3.-4.
11.
Sue
DW, Sue D. Counseling the culturally diverse: theory and practice. New York:
Wiley; 2003.
12.
Sue
DW, Sue D. Counselling the culturally different: Theory and practice. 2nd ed.
New York: John Wiley & Sons; 1990.
13.
Sue
DW, Arredondo P, McDavis RJ. Multicultural counseling competencies and
standards: a call to the profession. J Couns Dev. 1992;70:477.-486 doi:
10.1002/j.1556-6676.1992.tb01642.x.
14.
Campinha-Bacote
J. The process of cultural competence in the delivery of healthcare services: a
model of care. J Transcult Nurs. 2002;13(3):181.-184 doi:
10.1177/10459602013003003.
15.
Arredondo
P, Toporek R, Brown SP, Jones J, Locke D, Sanchez J. Operationalization of the
multicultural counseling competencies. J Multicult Couns Dev. 1996;24:42.-78
doi: 10.1002/j.2161-1912.1996.tb00288.x.
16.
Pinto-Meza
A, Moneta MV, Alonso J, Angermeyer MC, Bruffaerts R, Caldas de Almeida JM.
Social inequalities in mental health: results from the EU contribution to the
world mental health surveys initiative. Soc Psychiatry Psychiatr Epidemiol.
2013;48(2):173.-181 doi: 10.1007/s00127-012-0536-3.
17.
Monteiro
AP. Migration integration and internationalization of health care: immigration,
mental health of eastern European immigrants in Portugal, IMISCOE international
migration. Lisbon: Integration and Social Cohesion Conference; 2008.
18.
Freire
JE, Moleiro C, Farcas D, Pereira C, Pinto N, Roberto S, Cruz F, Ribeiro J.
Avaliação de competências para a diversidade individual e cultural:
desenvolvimento de uma medida comportamental para profissionais de saúde
mental. Migrações e interculturalidade. Porto: AGIR Associação para a
Investigação e Desenvolvimento; 2010;:241.-263.
19.
Monteiro
AP, Mendes AC. Multicultural care in nursing - from the theoretical paradigm to
the subjective experiences in clinical settings. Open J Nurs. 2013;3:557.-562
doi: 10.4236/ojn.2013.38076.
20.
Loftin
C, Hartin V, Branson M, Reyes H. Measures of Cultural Competence in Nurses: An
Integrative Review. The Scientific World Journal 2013; 4:
289101. Retrieved March 27th, 2014 from http://dx.doi.org/10.1155/2013/289101.
21.
Moleiro
C, Marques S, Pacheco P. Cultural diversity competencies in child and youth
care services in Portugal: development of two measures and a brief training
program. Child Youth Serv Rev. 2011;33(5):767.-773 doi:
10.1016/j.childyouth.2010.11.022.
22.
Khawaja
NG, Gomez IF, Turner G. Development of the multicultural mental health
awareness scale. Aust Psychol. 2008;44(2):1.-11.
23.
British
Psychological Society Ethics Guidelines for Internet-mediated Research .
Leicester: The British Psychological Society. 2003. Accessed 8 Jun
2013. Available in:
http://www.bps.org.uk/system/files/Public%20files/inf206-guidelines-for-internet-mediated-research.pdf.
24.
APA.
Ethical Principles of Psychologists and Code of Conduct. 2013. Accessed 8 Jun
2013. Available in: http://www.apa.org/ethics/code/.
25.
Campinha-Bacote
J. Delivering Patient-Centered Care in the Midst of a Cultural Conflict: The
Role of Cultural Competence OJIN: The Online Journal of Issues in
Nursing. 2011; 16 (2):5.DOI: 10.3912/OJIN.Vol16No02Man05
26.
Sue
DW, Bernier JE, Durran A, Feinberg L, Pedersen P, Smith EJ, Vasquez-Nuttall E.
Position paper: Cross-cultural counseling competencies. Couns Psychol.
1982;10:45.-52 doi: 10.1177/0011000082102008.
27.
Nunnally
JC, Bernstein IH. Psychometric theory. McGrawHill: New York, NY; 1994.
28.
Hill
MM, Hill A. Investigação por questionário. Edições Sílabo: Lisboa; 2000.
29.
George
D, Mallery P. SPSS for windows step by step: a simple guide and reference. 11.0
update (4th ed. Boston: Allyn & Bacon; 2003.
30.
Smith
WG. Does Gender Influence Online Survey Participation ?: A
Record-linkage Analysis of University Faculty Online Survey Response Behavior .
ERIC-Education Resources Information Center, 6 June 2008. Retrieved on Setember
28th, 2015. Available in:
http://scholarworks.sjsu.edu/cgi/viewcontent.cgi?article=1003&context=elementary_ed_pub.
No comments:
Post a Comment