research environment meaning

Research Environment

• In book: A Guide to Responsible Research (pp.1-17)
• This person is not on ResearchGate, or hasn't claimed this research yet.

Abstract and Figures

Discover the world's research

• 25+ million members
• 160+ million publication pages
• 2.3+ billion citations
• SCI ENG ETHICS

• Haiping Qiao
• David Moher

• ANNU REV PSYCHOL

• Science Staff
• SCHIZOPHRENIA BULL
• Bernard A Fischer
• Recruit researchers
• Join for free
• Login Email Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google Welcome back! Please log in. Email · Hint Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google No account? Sign up

Improving research environments

Creating a healthier future for everyone depends on thriving research environments that enable open, engaged, equitable, ethical and efficient research.

• Share with Facebook
• Share with X
• Share with LinkedIn
• Share with Email

Excellent research happens in environments where people from all backgrounds are treated with respect, supported and enabled to thrive.

It requires attention to ethical, social and cultural considerations, and engagement with the needs and perspectives of relevant communities.

We believe that creative and high-quality ideas must be open and accessible to everyone to drive innovation and achieve the most significant impact.

These values cut across everything we do and everyone we work with. They are at the heart of the positive and inclusive research culture we want to encourage with our funding so that:

• the research that we support is strengthened by being ethical, open and engaged
• the people we fund thrive in equitable, diverse and supportive research cultures
• Wellcome is known as an innovative, efficient and inclusive partner and organisation

By taking a holistic view of the environmental factors that impact research outcomes, Wellcome can achieve its ambition to be an inclusive funder of research to improve health for everyone.

Hannah Hope

Open Research Lead

Our approach  

Improving research environments is a priority that cuts across all of Wellcome’s funding teams, underpinning our work on  discovery research ,  climate and health ,  infectious disease  and  mental health . 

We also aim to contribute to the broader research ecosystem to ensure that Wellcome researchers have access to the resources, tools, and skills to pursue their work in ways that produce better evidence and meaningful impacts. This includes convening community events, doing policy work, investing in infrastructure, commissioning research and occasionally, offering funding for relevant activities.

Our activities aim to: 

• ensure that research is guided by and responsive to the needs and views of people involved in, or impacted by the work  
• navigate the complexities of innovation and encourage ethical, transformative research  
• openly share ideas, data and findings to speed up progress, enable reproducible research, and reduce duplication
• shape research culture and communities so that researchers from all backgrounds can thrive as they pursue bold ideas
• generate evidence and metascience on funding practices

Our work in action  

Europe pmc (pubmed central).

We are a funder of Europe PMC which provides an online database offering free access to published biomedical research.

Global Health Bioethics Network (GHBN)

We helped to establish the GHBN, which brings together bioethics scholars from across the world to collaborate on exciting ethics research and support early-career bioethicists.

National Coordinating Centre for Public Engagement (NCCPE)

The NCCPE promotes high-quality and equitable approaches to engagement across the knowledge sector. It is co-funded by Wellcome, Research England and UKRI.

Research on Research Institute (RoRI)

We support the RoRI, which generates insights on how to improve funding and assess innovative models for research funding.

Accelerator Awards

We launched this award to provide flexible funding for researchers of Black, Bangladeshi and Pakistani heritage in the UK to undertake activities that will help them reach their next career stage.

Institutional Fund for Research Culture (IFRC)

Wellcome's Institutional Fund for Research Culture supports initiatives to improve research culture at institutions across the UK and Ireland.

In2Research

We help fund this social mobility programme that supports people from low socio-economic backgrounds to progress to postgraduate research.

Investigating the effects of open sharing commitments

We commissioned a report to explore the impact of calls to rapidly and openly share Covid-19 research findings to inform public health response.

Research culture across Africa

We commissioned this research to understand what constitutes a “positive and inclusive” research culture in Africa.

How this applies to your research  

You can expect to find questions about research environment within the funding criteria for most of our awards. You should be able to demonstrate how your research is open, ethical and engaged, as well as describe how you will foster a positive and inclusive research culture.

Wellcome also has several research policies related to open and ethical research , and we recommend that researchers consult these when designing funding applications and delivering successful awards.

Looking for research funding?

Wellcome does not have specific funding for research environment; however, it is a theme in all research grant funding and is often a criterion in other procurement processes.

Dan O’Connor

Head of Research Environment

Carleigh Krubiner

Bioethics Lead

Shomari Lewis-Wilson

Senior Manager, Research Culture and Communities

If you have general enquiries or ideas related to our Research Environment work, please contact us on

[email protected]

Related content  

Diversity and inclusion: helping more ideas thrive

Let's reimagine how we work together

Open research

• Search Menu
• Sign in through your institution
• Advance articles
• Author Guidelines
• Submission Site
• Open Access
• Why Publish?
• About Research Evaluation
• Editorial Board
• Advertising and Corporate Services
• Journals Career Network
• Self-Archiving Policy
• Dispatch Dates
• Journals on Oxford Academic
• Books on Oxford Academic

Article Contents

The research excellence framework, topic modelling the research excellence framework, predicting environment scores, general discussion, acknowledgements, data availability, what is a high-quality research environment evidence from the uk’s research excellence framework.

• Article contents
• Figures & tables
• Supplementary Data

Matthew Inglis, Elizabeth Gadd, Elizabeth Stokoe, What is a high-quality research environment? Evidence from the UK’s research excellence framework, Research Evaluation , 2024;, rvae010, https://doi.org/10.1093/reseval/rvae010

• Permissions Icon Permissions

As part of the UK university sector’s performance-related research funding model, the ‘REF’ (Research Excellence Framework), each discipline-derived ‘Unit of Assessment’ must submit a statement to provide information about their environment, culture, and strategy for enabling research and impact. Our aim in this paper is to identify the topics on which these statements focus, and how topic variation predicts funding-relevant research environment quality profiles. Using latent Dirichlet allocation topic modelling, we analysed all 1888 disciplinary ‘unit-level’ environment statements from REF2021. Our model identified eight topics which collectively predicted a surprisingly large proportion—58.9%—of the variance in units’ environment scores, indicating that the way in which statements were written contributed substantially to the perceived quality of a unit’s research environment. Assessing research environments will increase in importance in the next REF exercise and the insights found through our analysis may support reflection and discussion about what it means to have a high-quality research environment.

For the past four decades, higher education institutions in the UK have been subject to evaluations of their research by the higher education funding councils. The first evaluation, the ‘Research Selectivity Exercise (RSE)’ took place in 1986 (for a history, see Bence and Oppenheim 2005 ). Over the years, and with six assessments between 1986–08, the RSE evolved into the Research Assessment Exercise (RAE) and then, in 2014, into the ‘Research Excellence Framework’ (REF). For each evaluation, university disciplines and fields of study were divided into ‘Units of Assessment’ (UoAs). The most recent assessment took place in 2021, with results published in 2022.

As well as name changes, the requirements for submissions have evolved (see Marques et al. 2017 ), from the “quick and dirty” ( Jones and Sizer 1990 ) approach taken in 1986 through to including/excluding particular categories of staff; changing the minimum/maximum numbers of publications per individual; the introduction of research environment statements (RAE 1996), and the introduction of impact case studies (REF 2014). Two constants about RSE/RAE/REF remain: the original principle of peer assessment, despite the rise of publication metrics in other domains, and the use of the results to distribute government funding.

The RSE represented the creation of “the first and most highly institutionalised research evaluation system worldwide” ( Marques et al., 2017 : 822). Since then, the RAE/REF has been widely discussed and used as a model for other countries (e.g. Geuna and Martin 2003 ) or resisted and rejected (e.g. Swedish Government 2016 ), but rarely adopted wholesale in countries internationally ( French, Massy and Young 2001 ; for overviews, see Sivertsen 2017 ; Thomas et al. 2020 ; Pinar and Horne 2022 ). Either way, the discourse of the RAE/REF reaches far beyond the UK.

Analysing the research excellence framework

Unsurprisingly, the RAE/REF has been scrutinized in terms of (i) critiques of the politics and methodologies that underpin the process and, (ii) quantitative and qualitative analyses of submissions, assessment processes, and results themselves. The former comprises a literature too vast to cover substantially here, but includes criticisms of the trend towards a competitive, neoliberal, and commodified higher education system (e.g. Fairclough, 1995 ; Brown and Carasso, 2013 ), of the impact of assessment on individual disciplines and interdisciplinarity (e.g. Pardo-Guerra 2022 ), and of unintended consequences (for overviews, see Gillies 2008 ; Brassington 2022 ; Pinar and Horne 2022 ; Watermeyer and Derrick 2022 ). The RAE/REF has driven both policy and debate in UK higher education, with a series of consultations, evaluations, recommendations, and iterated processes ( Manville et al. 2015 ; Curry, Gadd and Wilsdon 2022 ).

Another approach to evaluating and critiquing the RAE/REF focuses on the actual content of HEIs’ submissions to RAE/REF, using both quantitative and qualitative methods. Several of these studies have used similar text-mining or topic modelling and related methods to those used in this paper. Perhaps because of the increasing significance of research impact over the past decade ( Derrick and Samuel 2016 ; Kellard and Śliwa 2016 ; Sutton 2020 ; Jensen, Wong and Reed 2022 ), several studies have scrutinized the content and composition of case studies. For instance, a report commissioned by the UK research funding councils ( King’s College London and Digital Science 2015 ) used text-mining and qualitative analysis to provide an initial assessment of all REF2014 impact case studies, making observations about the diverse range of impacts, their underpinning research, and their global reach (see also Terämä et al. 2016 ). Reichard et al. (2020) conducted two studies using qualitative thematic and quantitative linguistic analysis of REF2014 impact case studies to identify what individual words and phrases were associated with high and low scores. They identified numerous “lexical bundles” associated with lower (e.g. “involved in”, “has been disseminated”, “the event”) and higher (e.g. “the government’s”, “in the UK”) scores, the former associated with describing activities and pathways to impact and the latter evidence of significant and far-reaching impact itself.

Within the wider literature on REF methodology and submissions, the least scrutinized aspect is the environment statement ( Thorpe et al. 2018a ). We now turn to discuss what we know and do not know about REF environment statements, starting by describing the submission requirements for the 2021 assessment.

The REF environment statement

The environment statement(s) and their submission requirements.

As noted above, an environment statement was introduced relatively early into the UK RAE/REF cycle. In 2021, two main changes occurred: the introduction of a pilot “Institutional-level environment statement”, and the removal of an ‘impact template’ from REF2014 and the incorporation of ‘research and impact’ as one environment statement in 2021.

The aim of REF2021 unit-level environment statements was to provide assessable information about each UoA’s “environment for research and enabling impact” (Guidance on Submissions, p. 82) and especially its “vitality” (“the extent to which a unit supports a thriving and inclusive research culture for all staff and research students, that is based on a clearly articulated strategy for research and enabling its impact, is engaged with the national and international research and user communities and is able to attract excellent postgraduate and postdoctoral researchers”) and “sustainability” (“the extent to which the research environment ensures the future health, diversity, wellbeing and wider contribution of the unit and the discipline(s), including investment in people and in infrastructure”) (Panel Criteria and Working Methods, p. 58). For REF 2021, detailed guidance notes and a template were provided to structure the information in four sections: (1) “unit context, research and impact strategy”; (2) “people, including: staffing strategy and staff development, research students, equality and diversity”; (3) “income, infrastructure and facilities”, and (4) “collaboration and contribution to the research base, economy and society.” The permitted length of the statement varied according to the number of staff in a UoA, from 8,000 (for a submission comprising 1–19.99 FTE) to 12,000 words (plus 800 further words per additional 20 FTE). The environment statement was worth 15% of the funding allocation. For Main Panels A, B, and C, each of the four subsections attracted equal weighting; in Main Panel D, sections (1) and (4) attracted 25%, the ‘People’ section 30%, and ‘income, infrastructure and facilities’ 20%.

Analyses of environment statements

To the best of our knowledge, no close analyses of REF2021 environment statements have yet been published, apart from Manville et al.’s (2021) real-time study of REF as it happened, including focus group research on academic and professional service staff experiences of completing them. However, research has been conducted on REF2014 environment statements. For example, Matthews and Kotzee (2022) analysed both REF (2014 submission) and TEF (Teaching Excellence Framework, 2017 submission) documentation with the aim of investigating links between research and teaching. They found that the term “research-led”, analysed in the context of its collocates, was often used in connection with teaching, and argued that “according to what universities themselves write in institutional texts, teaching and research are not always in a mutually beneficial entanglement, but often rather a one-way relationship in which research expertise and institutional prestige are used to bolster claims of teaching excellence” (p. 578).

Mellors-Bourne, Metcalfe and Gill (2017) also used text-mining to assess the level of engagement with equality and diversity in the ‘People’ section of REF2014 environment statements. This included evidence of participation in schemes such as ‘Athena Swan’ and Stonewall’s, and the relative frequencies of words pertaining to the topic: ‘equality’, ‘diversity’, ‘Athena’, ‘gender’, and ‘ethnicity’. Mellors-Bourne et al. found that statements focused predominantly on gender; that “the word ‘equality’ was used on average between once and twice within each environment statement” (p. 2), and that the Main Panels A and B (the STEM disciplines) mentioned ‘Athena’ more than twice as much as Main Panels C and D. The researchers also found “[e]vidence suggesting a positive relationship between REF research environment subprofiles (scores) and reference to key E&D terms within submissions, overall and at the level of the main panels” (p. 2). In REF2021, the focus on equality, diversity and inclusion in the guidance notes extended well beyond the ‘People’ section, presumably to encourage HEIs to demonstrate how EDI strategies and outcomes were embedded in all areas of research and impact.

Thorpe et al. (2018a , 2018b ) used computer-assisted text analysis to scrutinize the language content and ‘tone’ of REF2014 environment statements in business and management schools. They sought to understand whether the way environment statements were written differed between high and low scoring universities. They found that higher-ranked universities used less passive voice, were more coherent, adopted “a ‘finished article’ discourse rather than a ‘we are developing’ discourse”, cited “specifics rather than generalities”, and were more self-referential ( Thorpe et al., 2018a , p. 582). In terms of tone, the authors found that, perhaps counterintuitively, higher-scoring statements scored lower in terms of ‘activity’ tone “that evokes a ‘safe’, ‘staid’, ‘orthodox’, ‘conservative’, and ‘settled’ environment that is not disturbed (unduly at least) by reform, disruption, or major staff turnover” ( Thorpe et al. 2018b : 60). The authors concluded that “low-ranked universities could have achieved higher scores by reflecting on particular areas of word choice and the potential effects of those choices on assessors” ( Thorpe et al. 2018b : 53).

Like Reichard et al.’s (2020) analysis of impact case studies, Thorpe et al. (2018a , 2018b ) revealed the importance of language as well as content in the production of environment statements. They concluded that “the accompanying narrative played an important role in determining REF2014 environment scores” (2018a: 572). Furthermore, in contrast to impact case studies, which included corroborating evidence as part of submission, “little supporting evidence was required in environment submissions”, meaning that “there is potential for writing quality to have an even larger effect in environment submissions, and for HEIs to use language-related techniques to manage their image” ( Thorpe et al. 2018a : 574).

In June 2023 the ‘Future Research Assessment Programme’ (FRAP) published their Initial Decisions on REF 2028 ( Joint UK HE Funding Bodies 2023 ), although it has since been announced that the exercise will be delayed until 2029. Despite the Institution-Level Environment Panel Pilot Panel (Research England 2022) recommending the removal of unit-level environment statements and focusing instead on institution-level statements, the Initial Decisions propose the retention and assessment of both. In the 2029 exercise, ‘Environment Statements’ will be broadened to become ‘People, Culture & Environment (PCE) Statements’ with a concomitant increase in weighting from 15% to 25%. The Decisions state that “the collection of evidence for the people, culture and environment element will move towards a more tightly defined, questionnaire-style template that will create greater consistency across submissions and focus on demonstrable outcomes ( Joint UK HE Funding Bodies 2023 )”. At the time of writing this is yet to be developed and it is unclear what proportion of the submission will take a narrative format and what proportion will comprise data and evidence. However, given the qualitative nature of the dimensions being assessed, it is likely that there will be a significant narrative element. Furthermore, given the increased weighting of PCE, any such element should have an even greater bearing on overall results.

In sum, the focus on environment statements as a subset of research on the UK REF has, to date, been small. It has either been partial (e.g. has analysed just one UoA ( Thorpe et al. 2018a , 2018b )) or not focused on the most recent exercise. The aim of this paper is to investigate whether thematic patterns may be identified in the 2021 submissions, whether such patterns may be correlated with scores, and what this might teach us about crafting future environment narratives. We begin by introducing the method we adopted, latent Dirichlet allocation topic modelling.

Topic modelling is a computation method that seeks to analyse the content of many texts by identifying a small number of semantically connected themes or topics ( Blei, Ng and Jordan 2003 ). The aim is to take a collection of unstructured texts and identify the topics they cover by studying their words. For example, if a document uses the words ‘water’, ‘sand’, and ‘swimming’ with an unusually high frequency, this may constitute evidence that the document is about beaches. One way to understand the topic modelling approach is to think about how to create documents from a predefined set of topics, defined to be probability distributions over words. For instance, our beach topic might assign very high probabilities to ‘water’, ‘sand’ and ‘swimming’, medium probabilities to ‘inflatable’, ‘spade’ and ‘picnic’, and low probabilities to ‘dioxide’, ‘carpentry’ and ‘veneer’. Similarly, we might define a topic about Greece (which might perhaps have a high probability associated with the words ‘Greek’, ‘Athens’, ‘souvlaki’ and so on). If we wanted to create a new document about beach holidays in Greece, we might choose 30% of the new documents words to be from the beach topic, 30% from the Greece topic, perhaps 30% from a travel topic, and 10% from other topics. A document about Greek beach holidays can then be created by simply sampling words from each topic using the appropriate probabilities. This method makes two simplifying assumptions. First, the ‘bag of words’ model of text is adopted by ignoring word order; and second, so-called ‘stop words’ (words such as ‘the’, which is topic independent) are ignored.

The topic modelling approach can be thought of as carrying out this document construction process in reverse. We start with a large collection of texts, assume that they were created in this way, and then computationally identify the topics that would have been most likely to lead to these documents using a latent Dirichlet allocation (LDA) algorithm ( Blei, Ng and Jordan 2003 ; Grimmer and Stewart 2013 ). Topic modelling adopts a grounded theory mentality: the analyst has no preconceived ideas about what topics will be identified, instead topics/themes emerge from the analysis process. Once topics have been identified, the semantic content of each document can be analysed by studying the topic composition of each document. For instance, we may find that Document 1 contains 6% of words from Topic 1, 30% from Topic 2, 0% from Topic 3, and so on.

We downloaded all 1888 unit-level environment statements (a total of 18.0 m words) from the REF website. These were converted from pdf into plain text using the UNIX pdftotext command ( Poppler 2022 ). We used MALLET (version 2.0.8RC2), a UNIX topic modelling tool ( McCallum 2002 ), to calculate possible models, using MALLET’s default list of stop words.

Topic modelling requires that one specifies how many topics the LDA algorithm should identify. By making different choices researchers can specify the granularity of their analysis. We adopted the perplexity approach to decide on the number of topics. Each model can be assigned a perplexity, which is analogous to a model fit ( Blei, Ng and Jordan 2003 ). Perplexity can be calculated by fitting a model with a specified number of topics to a subset of the documents, and then assessing its fit to the remaining documents. One can always reduce the perplexity (or increase the fit) by fitting a model with a larger number of topics, although at some point the benefit of doing so will be offset by the increased difficulty of interpretation. Jacobi, van Atteveldt and Welbers (2016) suggested using a method similar to the scree test often used in factor analyses: by calculating the perplexity of models with a range of different topic numbers, it is possible to determine if there is a point at which the benefit, in terms of reduced perplexity, of increasing the number of topics appears to level off.

We split the environment statements into a training corpus (80% of statements) and a testing corpus (20% of statements), fitted models with 10, 20, 30, …, 100 topics to the training corpus, and calculated the associated perplexities using the testing corpus. These perplexity figures are shown in Figure 1 . We then fitted a piecewise linear regression to these points, which suggested that the ‘elbow’ of the graph appeared at 41.99 topics. We therefore selected 42 topics for our main analysis.

Perplexities associated with models with 10, 20, 30, …, 100 topics. The dotted lines show a piecewise linear regression line of best fit.

Topic modelling has an important advantage over more traditional qualitative analytical techniques in that it is extremely inclusive. Given that 1888 unit-level environment statements were returned to REF2021, containing ∼18 m words, it would have been impractical for a human analyst to read and analyse each statement. However, topic modelling is not purely quantitative: the LDA algorithm identifies topics which must then be interpreted. One common approach to this task is to conduct careful qualitative analyses of documents that contain a high proportion of words from each topic. We return to this issue later in the paper.

The 42 topics identified are shown in Table 1 . The table shows the characteristic words associated with each topic, the statement with the highest proportion of words from that topic, and the label we gave the topic. These labels were based on our interpretations of studying the characteristic words, the statements with particularly high proportions of words from the topic, and the statements with particularly low proportions of words from the topic.

Descriptive statistics, averaged across each of the four quarters of the experiment, for each of the four indices under consideration

Of the 42 topics, 28 were disciplinary specific. For example, Topic 42 was characterised by words including ‘clinical’, ‘cancer’, ‘medicine’, ‘disease’, ‘MRC’ and ‘NHS’, and the top 10 environment statements in terms of the proportion of words with this topic were all returned to the Clinical Medicine panel. We therefore labelled this topic ‘Clinical Medicine’. In some cases our model combined two or more disciplines. For instance, Topic 2 was characterised by the words ‘philosophy’, ‘religion’, ‘theology’, ‘religious’ and ‘ethics’. Of the top 20 statements with high proportions of words from this topic, 9 were returned to the Philosophy panel and 11 to the Theology and Religious Studies panel. We used the label ‘Philosophy and Religion’.

There were five geographical topics. For instance, Topic 8 was characterised by the words ‘Liverpool’, ‘Leeds’, ‘Manchester’, ‘York’, ‘Sheffield’ and ‘Yorkshire’, all cities/regions in the north of England, and the statements with the highest proportion of words from this topic were from northern universities. There were geographical topics associated with the North, the West Country, Scotland, London and Wales.

We also found a topic, Topic 25, that was used by institutions that organise their academic work through constituent colleges. The topic was characterised by words including ‘university’, ‘faculty’, ‘centre’, ‘college’ and ‘institute’, as well as geographical terms that referenced multi-college universities (‘London’, ‘Oxford’, and ‘Cambridge’). Of the 25 statements with the highest proportions from this topic, 23 were returned by constitute faculties or colleges of the University of Oxford, the University of Cambridge or the University of London. The exceptions were statements from Kingston University (an institution based in London) and Oxford Brookes University (an institution based in Oxford).

Of most interest for our purposes are the remaining eight topics. To identify appropriate labels we followed a similar process, separately for each topic. First, we studied the topic’s characterising words. Second, we read the five environment statements with the highest proportion of words from the topic, and the five environment statements with the lowest proportion of words from the topic. Finally, we conducted concordance analyses to identify how characterising words were used in statements with high and low proportions of words from the topic. This involved using a keyword in context (KWIC) tool from a traditional corpus linguistics package ( Anthony 2022 ). For instance, if ‘faculty’ was a characteristic word for a topic, we would find every occurrence of ‘faculty’ in the five statements with the highest proportion of words from the topic and read the surrounding context to identify how the word was typically being used. We would then do the same for the five statements with the lowest proportion of words from the topic. To illustrate our approach, we first discuss the reasoning behind our naming of Topic 16 in some detail, and then discuss each of the remaining seven topics in turn. Note that the online data associated with this article includes the topic weightings derived from our model for each topic and each environment statement submitted to REF2021, so interested readers can independently verify our analyses and assess for themselves the topic names’ appropriateness.

Although all REF environment statements are publicly available from the Research England website, we opted to redact individuals’ names in the quotes reported below as they are not relevant to the research questions we asked. We have, however, not anonymised at the institution or department level, as these may assist readers interpret the topics.

Topic 16—Immature Research Environment

The proportion of words from Topic 16 used by statements varied from 0.0001 (Imperial College’s Mathematical Sciences statement) to 0.362 (Bedfordshire’s Business and Management Studies statement). The topic was characterised by words such as ‘research’, ‘university’, ‘staff’, ‘international’, ‘REF’, ‘member’, and ‘members’ (see Table 1 ). Unlike with some of the other topics discussed below, we did not find these words very insightful for determining the semantic content of the topic.

Our next step was to carefully read the five statements with the highest proportion of words from the topic and the five statements with the lowest proportion of words from the topic. This revealed that the topic-defining words seemed to be being used in characteristic ways by those statements with a high proportion of words from the topic. Specifically, Topic 16 was characterised by descriptions of how the units were trying to encourage staff to engage in research. For example, the Wrexham Glyndŵr University Computer Science and Informatics statement (Topic 16 proportion 0.327) noted that “Data from October 2020 indicates that 38% of the 13 members of academic staff associated with UoA11 [the Computer Science and Informatics unit] have a doctoral qualification” and that “An encouraging sign is that 38% of UoA11 staff are studying towards a doctorate.” Similarly, the Liverpool John Moores University Business and Management Studies statement (T16 proportion 0.339) noted that 31 members of staff “are being supported in their research-related activities, with a view to them being research active in the next assessment period” and that staff were supported by holding seminars, where the invited external speakers were “editors of peer-reviewed journals with high impact factors” in order to assist staff “target publications in highly respected journals”. The Bedfordshire Business and Management Studies statement (T16 proportion 0.362) emphasised that “Staff members are strongly encouraged to attend international conferences and present their research results” and that their staff “are allocated dedicated research time as part of their workload”. In contrast, the statements with low frequencies of words from Topic 16 seemed to take for granted that their academic staff had doctorates and routinely conducted research.

Next, we conducted concordance analyses comparing how the topic’s defining words were used in the five statements with a high proportion of words from Topic 16 with how they were used in the five statements with a low proportion of words from the topic. For instance, we compared how these statements used the word “research”. In the five high statements there were 19 uses of “research active” (e.g. “continued to be research active”, “support to be research active”, “increased the number of research active staff”, “sought to retain research active staff”, “staff on the cusp of being research-active”) compared to just one in the five low statements, which appeared in a subheading in the University College London’s Law statement (“2.1 Research-active staff and output selection profile”).

As another example, across the five high Topic 16 statements there were 82 instances of “conference”, including numerous examples of conferences that had been attended by staff from these units. In contrast, this word appeared only 29 times in the five low Topic 16 statements and tended to be used as an illustration of a wider point. For example, in the Cambridge philosophy statement (T16 topic proportion 0.00003), the organisation of the Cambridge Platonism conference was given as an example of the unit’s interdisciplinary research (the conference was jointly organised with the Cambridge Faculty of Divinity).

To give one final example, there was also a difference in the way the high- and low-T16-proportion statements used the word “journal”. The five high-T16-proportion statements contained 37 instances of this word. In some cases, these were examples of how members of the unit had written research articles, e.g. Newman University’s Sport and Exercise Sciences, Leisure and Tourism statement (T16 proportion 0.360) noted that “Visiting Professor [anonymised] has produced a manuscript currently in review in the European Respiratory Journal Open”. In other cases these were lists of interactions with journals: Wrexham Glyndŵr University’s Computer Science and Informatics statement explained how one colleague was “on the review panel for a further 6 journals”. In contrast, the five low-T16-proportion statements had only 13 instances of the word ‘journal’. These tended to be examples of how the unit was contributing to the wider academic community. For instance, the University College London law environment statement (T16 proportion 0.0003) discussed how they were progressing towards an open research environment and exemplified this by noting how one member of the unit had “founded Europe and the World: A Law Review as a fully peer-reviewed OA [open access] journal”.

In sum, we concluded that those statements which had a high proportion of words from Topic 16 tended to spend a large proportion of their statement discussing how they were attempting to encourage or support routine research activities. These kinds of discussions were absent from those statements with a low proportion of words from this topic. We therefore named this topic “Immature Research Environment”.

Topic 4—Internal Structure of Research Units

Topic 4 was characterised by the high use of words such as ‘unit’, ‘unit’s’, ‘faculty’, ‘section’, ‘themes’, ‘theme’ and ‘institutional’. The proportion of words from this topic ranged from 0.000 (the University of Edinburgh’s Clinical Medicine statement) to 0.150 (the University of Cumbria’s Business and Management Studies statement). The topic tended to be characterised by detailed descriptions of the internal structure of the units. For instance, the University of Cumbria’s Business and Management Studies statement (T4 proportion 0.150) devoted 1.5 pages of their statement to the “unit context and structure” which noted how, during the assessment period, they had created a new institute and developed three new research themes. Similarly, the University of Winchester’s English Language and Literature statement (T4 proportion 0.137) spent just over a page discussing their unit context and structure, noting how the unit was situated within the University’s department and faculty structure, how it contained a research centre, and how the Centre interacted with other centres across the University. We named this topic “Internal Structure of Research Units”.

Topic 7—Career Development and EDI

Topic 7 was characterised by words such as ‘staff’, ‘support’, ‘training’, ‘including’, ‘access’, ‘career’ and ‘diversity’. The proportion of words from this topic ranged from 0.004 (The Royal Agricultural University’s Agriculture, Food and Veterinary Sciences statement) to 0.399 (Leeds Arts University’s Music, Drama, Dance, Performing Arts, Film and Screen Studies statement). Statements with a high proportion of words from Topic 7 tended to have long sections that discussed how the unit supported staff and student development, and about their equality and diversity processes. For instance, the University of Nottingham’s Politics and International Studies statement (T7 proportion 0.290) included careful statistical analyses of their gender balance at different career stages, as well as analyses of their staff by ethnicity, disability and age profiles. The statement then went on to discuss how these analyses informed “EDI-focused improvements”. For instance, in response to “too few members from underrepresented groups in leadership roles” the statement noted how the unit had reconstituted its EDI committee and “increased leadership by women in major committees”. In contrast, the Royal Agricultural College’s Agriculture, Food and Veterinary Sciences statement (T7 proportion 0.004) devoted just 50 words to EDI issues, and only used the word ‘diversity’ in the context of their research on “the global distribution of earthworm diversity”. We named this topic “Career Development and EDI”.

Topic 18—Staff Ways of Working

Topic 18 was characterised by words such as ‘work’, ‘school’, ‘colleagues’, ‘teaching’, ‘group’, ‘members’, ‘part’ and ‘years’. Some statements contained a very low proportion of words from this topic, e.g. Heriot-Watt University’s Architecture, Built Environment and Planning statement (T18 proportion 0.000), whereas others contained a substantial proportion from it, e.g. the University of East Anglia’s Law statement (T18 proportion 0.262). The statements with a high proportion of words from the topic were characterised by many concrete descriptions of staff working practices. For example, the University of Newcastle upon Tyne’s Classics statement (T18 proportion 0.244) described how “Members who have held a substantial administrative role are entitled to an extra semester of research leave”. Similarly, the University of St Andrews’s Economics and Econometrics statement (T18 proportion 0.255) discussed the process by which academic staff can apply for sabbatical leave: “The HoS considers applications in relation to the general workload allocation process and, if there are doubts about the feasibility of accommodating all applications, the HoS consults a panel of senior colleagues.”

In our concordance analysis we noted that ‘work’ was commonly used as a verb in high-T18-proportion statements to describe concrete examples of how the unit operated (“the University granted [anonymised] two years of unpaid leave to work and develop Impact at the European Central Bank”), whereas in low-T18-proportion statements it was often used as a noun (“our work on public health engineering”). We also found that ‘members’ was more often used in high-T18-proportion statements to describe concrete internal activities (“Individual staff members can request travel funding and leave to attend masterclasses and short courses”), whereas in low-T18-proportion statements it was typically used to describe esteem activities (“Our researchers also chaired or have served as members of important grant panels”). We named this topic “Staff Ways of Working”.

Topic 28—REF-Focused Research Strategy

Topic 28 was characterised by words such as ‘UoA’, ‘REF’, ‘UoA’s’, ‘UoAs’, ‘section’, ‘cycle’ and ‘submitted’. The proportion of words from this topic varied from 0.000 (the University of Edinburgh’s Clinical Medicine statement) to 0.126 (the University of Winchester’s History statement). Statements with a high proportion of words from Topic 28 tended to use REF terminology to describe their research environment. For example, they might characterise their internal structure in terms of UoAs or ‘units’ rather than departments, centres or institutes. For instance, the University of Worcester’s English Language and Literature statement (T28 proportion 0.126) described “The Unit’s strategic research objectives”, “the unit’s impact strategy” and “the unit team”; and the University of Winchester’s History statement (T28 proportion 0.126) described how “the UoA had a devolved budget”, the existence of a “UoA working group” and “the strategic aims of the UoA over the cycle”. In contrast, the joint engineering statement from the University of Edinburgh and Heriot-Watt University (T28 proportion 0.000) contained no instances of ‘UoA’, and only used “unit” in the generic text used in the page header (“unit-level environment template (REF5b)”). Instead, they described how their research was organised into “cross-cutting organisational themes” and “interdisciplinary global research challenge areas”. Similarly, the University College London education statement (T28 proportion 0.000) discussed how their research was organised into departments and research centres, and only used the word ‘UoA’ in a table reporting the submission’s demographic data. We named this topic “REF-Focused Research Strategy”.

Topic 30—Exemplification of Strategy and Processes

Words that characterised Topic 30 included “e.g”, “including”, “funding”, “supported”, “grant” “PGRs”, “impact” and “awards”. Like Topic 16, it was not immediately obvious to us from studying these words what the topic referred to. However, when we compared the high-T30-proportion statements and the low-T30-proportion statements, we concluded that the topic was capturing an increased use of concrete examples to illustrate strategies and processes. To illustrate, the five statements with the highest proportion of words from Topic 30 made liberal use of “e.g.” to give explicit examples of the research strategies being described. For example, the University of Nottingham’s Geography and Environmental Studies statement (T30 proportion 0.278) described how they enable and facilitate impact: “the new Institutional Institute of Policy and Engagement has helped fund pump-priming engagement work (e.g. [anonymised]); fund high-level policy relevant talks (e.g. [anonymised] at Asia House and Chatham House) and aid development of policy briefs (e.g. [anonymised] on water management in the Red River, French on indebtedness and financial exclusion).” In their section on open research, they wrote that “The School developed and hosts online, openly accessible maps, including the Blue-Green Cities multiple benefits toolbox ([anonymised]) and the ‘black presences and the legacies of slavery and colonialism’ online map ([anonymised])”. Similarly, the University of Leicester’s Communication, Cultural and Media Studies, Library and Information Management statement (T30 proportion 0.266) noted that their “strategy of enabling researcher development in Media focuses on supporting ECRs and mid-career academics, to achieve external funding success. For example, 23 of the 40 awards secured in the REF period were to Assistant Professors.”

While low-T30-proportion statements also used exemplification, these tended to be less related to the research strategies and processes described in the statements. For instance, there were 22 instances of “e.g.” in Imperial College London’s Clinical Medicine statement (T30 proportion 0.000), of which 12 were used in front of lists of journals (“Our reach is demonstrated by publishing in specialist journals (e.g. Lancet Infect Dis [11], Nature Immunology [7]”) and a further 4 were used in front of scientific concepts (“…to reveal mechanisms of cardiovascular disease. e.g. identification of titin variants in health and disease”).

We named Topic 30 “Exemplification of Strategy and Processes”.

Topic 34—Industry Partners and Funding

Topic 34 was characterised by words such as ‘award’, ‘awards’, ‘industry’, ‘society’, ‘data’, and ‘international’. The proportion of words from this topic in statements varied from 0.000 (the University of East Anglia’s Area Studies statement) to 0.259 (the University of Bristol’s chemistry statement). Those statements which had a high proportion of words from Topic 34 devoted considerable space to discussing their industrial partnerships and research funding. For example, the Imperial College London Chemistry statement (T34 proportion 0.252) noted that “Collaborations with industry include GSK and Pfizer”, and that “members are involved in industry collaborations e.g. a £3.2M EPSRC BP Prosperity Partnership”. The University of Surrey Physics statement (T34 proportion 0.249) argued that their “world-class research is evidenced by grant awards over the REF period that total more than £19.6 million” and noted that they “work closely with industrial partners”. Given this focus, it was unsurprising that the correlation between the proportion of an environment statement made up of words from Topic 34 was strongly correlated with a submission’s research funding per FTE, r = 0.642, P < 0.001. Notably, however, this correlation was much reduced if research income was standardised within each UoA (to r = 0.275, P < 0.001). In other words, Topic 34 related to overall unstandardised research funding, meaning that statements with a particularly high proportion of Topic 34 words tended to come from highly funded scientific disciplines. Indeed, the mean proportion of words from Topic 34 for statements to Main Panels A (medicine, health and life sciences), B (physical sciences, engineering and mathematics), C (social sciences) and D (arts and humanities) were 0.109, 0.139, 0.044 and 0.020 respectively. In other words, statements from scientific disciplines tended to use more words from Topic 34 than statements from non-scientific disciplines, an observation consistent with our conclusion that the topic concerned industrial partnerships and funding. We named this topic “Industry Partners and Funding”.

Topic 40—Early Career Researcher (ECR) Development

The last of our eight general topics was Topic 40. This topic was characterised by words such as ‘development’, ‘develop’, ‘support’, ‘research’, ‘researchers’, ‘strategy’, ‘strategic’, ‘work’, ‘working’ and ‘funding’. Of the eight general topics, Topic 40 was the most common: on average environment statements devoted 18.6% of their content to it, although the proportions of words from Topic 40 ranged from 0.038 (Kingston University’s Philosophy statement) to 0.403 (Canterbury Christ Church University’s Sport and Exercise Sciences, Leisure and Tourism statement). Those statements with a high proportion of words from Topic 40 spoke at length about researcher development, with a particular focus on early career researchers. For instance, the Queen Margaret University Edinburgh Sociology statement (T40 proportion 0.360) discussed how they “support researchers in exploring and preparing for a diversity of careers, for example, through the use of mentors and careers professionals, training, and secondment” and the Solent University Southampton Sport and Exercise Sciences, Leisure and Tourism statement mentioned that a “research mentoring programme organised through [the School Advisory Group for Research] has been implemented to support researchers”. The five statements with the highest proportion of words from Topic 40 made 23 references to the Concordat to Support the Career Development of Researchers compared to 4 references in the five statements with the lowest proportion of words from this topic. We named the topic “Early Career Research (ECR) Development”.

For our main analysis, we asked whether the eight general topics that environment statements focused on were related to the quality profiles they received. Recall that panels assessed each submission’s environment using a five-point scale from ‘unclassified’ (“an environment that is not conducive to producing research of nationally recognised quality or enabling impact of reach and significance”) through to ‘4*’ (“an environment that is conducive to producing research of world-leading quality and enabling outstanding impact, in terms of its vitality and sustainability”). Each submission was awarded a ‘quality profile’ based on its environment statement and associated data (discussed below). For instance, the Open University’s submission to the Classics UoA was rated as having an environment where 25% of activity was 4* (world-leading), 50% was 3* (‘internationally excellent’), 25% was 2* (‘recognised internationally’) and 0% was 1* (‘recognised nationally’) or unclassified. For each submission we calculated a grade point average (GPA), which was a simple linear combination of the percentage of each quality level. So the Open University’s Classics submission obtained an environment GPA of 3.0 (0.25 × 4 + 0.5 × 3 + 0.25 × 2 + 0 × 1 + 0 × 0).

Alongside environment statements, the assessment panels were also provided with additional metrics associated with each submission. These included the full-time equivalent number of staff (FTE) being returned in the submission, the grant income that the unit had received during the assessment period (which could be broken down by source and date), and the number of doctoral degrees that the unit had awarded during the assessment period.

We ran a hierarchical regression predicting each unit’s environment GPA. In the first block we entered each unit’s FTE, their research income per FTE, and the number of doctoral degrees awarded per FTE. Each of these metrics was standardised (using z scores) within each UoA to take account of disciplinary norms (for instance, the mean grant income per FTE in the Clinical Medicine unit was £3.7 m compared to £74k in the English Language and Literature unit). In the second block we entered the proportion of each environment statement from the eight general topics discussed above.

The results of this regression are shown in Table 2 . Together the environment metrics could explain 47.3% of the variance in environment GPAs. When the topic weightings were added, an additional 21.9% of the variance could be explained, bringing the overall R 2 to 69.1%. Thus the weightings of these eight topics explained significant extra variance in environment GPAs, F (8, 1870) = 166, P < 0.001. When the eight topic weightings were used as predictors in the first block (ie before the metrics were entered) they explained 58.9% of the variance in environment GPAs, F (8, 1873) = 336, P < 0.001. In sum, the weightings associated with the eight general topics in our topic model predicted a surprisingly large proportion of the variance in submissions’ environment GPAs, indicating that the topics that environment statements focused upon made a substantial contribution to the perceived quality of each submission’s research environment.

A hierarchical regression analysis predicting environment GPA with various metrics (entered in Block 1) and topic weightings from the eight general topics (entered in Block 2)

P < 0.05.

P < 0.01.

P < 0.001.

As shown in Table 2 , of the eight topics, four were significant negative predictors of environment GPA, two were significant positive predictors and two were not significant predictors. Statements that had higher weightings from the Immature Research Environment, Staff Ways of Working, REF-Focused Research Strategy, and ECR Development topics were associated with lower environment GPAs. Statements that had higher weightings from the Exemplification of Strategy and Processes, and Industry Partners and Funding topics were associated with higher environment GPAs.

Because this regression analysis only assessed whether topic weightings were linearly associated with environment GPAs, we also investigated whether there were nonlinear relationships by inspecting scatterplots of topic weightings against environment GPAs separately for each topic. These are shown in Figure 2 , together with cubics of best fit. There appeared to be a clearly nonlinear relationship between topic weighting and environment GPA for Topic 7 Career Development & EDI. Placing little emphasis on this topic was associated with receiving a low environment GPA, but so was placing too much emphasis on it. The cubic of best fit obtained its maximum when 13.4% of the statement was made up of words from Topic 7 (recall that this figure is a percentage of all words in the statement, after stop words have been removed). Environment statements varied substantially in the extent that they discussed Career Development and EDI—the statement with the lowest emphasis on this issue had just 0.4% of its words from the topic, the statement with the highest had 39.9%. But the highest environment GPAs, on average, were obtained by statements where 13–14% of the statement focused on Career Development and EDI.

Scatterplots showing topic weightings (proportion of each statement made up of words from the given topic) against environment GPA, separately for the eight general topics. Bold lines are cubics of best fit.

Topic 40 ECR Development also showed a possibly nonlinear relationship between topic weighting and GPA, although this was less clearly the case than for Topic 7. For statements where between 0% and 20% of their words came from Topic 40 there was a reasonably flat relationship with GPA. But those statements with higher proportions from this topic showed a negative relationship between topic weighting and GPA.

Next, we explored whether environment statements that discussed more discipline-specific issues scored more highly than those which did not. In other words, we asked whether environment statements returned to, say, the Clinical Medicine UoA scored more highly when if they used more words from the Clinical Medicine topic. To investigate this we calculated the correlation between environment GPA and topic weighting for the disciplinary topic, separately for each UoA. These results are shown in Table 3 . While a large majority of these correlations were positive, indicating that environment statements that contained more discipline-specific language tended to score higher, there were systematic differences between broad subject areas. The mean correlations between the percentage of discipline-specific language and GPA for Main Panels A (medicine, health and life sciences), B (physical sciences, engineering and mathematics), C (social sciences) and D (arts and humanities) respectively were 0.472, 0.217, 0.172 and 0.093 respectively. For all main panels other than D, these means were significantly greater than zero.

Correlations between environment GPA and disciplinary topic weightings, per UoA (e.g. within the Clinical Medicine UoA, the correlation between environment GPA and weightings on Topic 42 was r = 0.560)

To compare the strength of the association between the extent to which submissions discussed disciplinary issues and their GPAs with the strength of the associations between the eight general topics discussed above and GPAs, we ran a regression analysis on submissions to the Business and Management Studies panel. We chose Business and Management as it was the panel which received the largest number of submissions (108), and so offered the greatest statistical power for an analysis of this kind. In this regression we used the eight general topics, plus the Business and Management topic (Topic 20) to predict environment GPAs. This model is shown in Table 4 . Crucially, the Business and Management topic weighting variable was a significant predictor in this model and had a standardised regression coefficient of β = 0.120, larger than that associated with Exemplification of Strategy and Process (Topic 30, β = 0.102), and roughly a third the size of Industry Partners and Funding (Topic 40, β = 0.336). In other words, using language associated with business and management was a stronger predictor of environment GPAs than giving examples of the unit’s strategy, and around a third as strong a predictor as discussing external funding and industrial partnerships.

A regression analysis predicting environment GPA of submissions to the Business and Management Studies panel, with the topic weightings from the eight general topics and the Business and Management topic

In sum, the more an environment statement included content from the relevant discipline, the higher the environment GPA it received, although this effect was more pronounced for medicine, health and life sciences, and less pronounced for the arts and humanities. To illustrate this, we analysed statements with the most and the least discipline-specific language from the ‘Biological Sciences’ and ‘Economics and Econometrics’ panels (the two panels where the relationship between discipline-specific language use and environment GPA was strongest). The differing content and emphasis were clear. For example, University College London’s Economics and Econometrics statement began with a sentence that listed its research strengths in “microeconomics, macroeconomics and econometrics” and went on to link their research focus to “the most pressing national and international socio-economic challenges of our time, such as inequality, migration, globalization, and sustainable growth” as well as international economic policy. By contrast, the opening remarks in the University of Northampton’s Economics statement focused on being a first-time submission, and noted that this new development “has been led in part by structural changes at the University level, but more significantly by the appointment of a new Dean.” Their first paragraph continued to list internal structure rather than discipline-relevant topics of strength and expertise.

Similarly, in the opening paragraph of Birkbeck’s submission to the Biological Sciences UoA, discipline-specific language was used to articulate the “fundamental biological questions” conducted by staff “who are using microbial, plant and animal systems to advance our understanding of the fundamental principles underlying molecular and cellular function, physiology and behaviour” In contrast, the opening paragraphs of the University of Worcester’s Biological Sciences statement described their first-time submission to the panel and articulated their internal structures and strategies (e.g. “the University went through an academic restructure introducing Colleges and Schools”; “The University Research Strategy 2014–19 outlined the key role played by Research Groups in operationalising plans and ambitions for excellent research”). Thus, right from the start of these statements, a focus on disciplinary contribution was much clearer in the higher-scoring submissions than those with a lower GPA.

Summary of main findings

We asked whether the perceived quality of a research environment, as measured in the UK’s Research Excellence Framework, could be predicted by the text used by that unit to describe their environment. By topic modelling the full text of all 1888 unit-level environment statements submitted to REF2021, we settled on a model that included eight specific topics that were distinct from disciplinary or geographical topics. These were related to the Internal Structure of Research Units, Career Development and EDI, Immature Research Environments, Staff Ways of Working, REF-Focused Research Strategies, Exemplifications of Staff Ways of Working, Industry Partners and Funding, and ECR Development. The proportion of words each statement included from these eight topics was surprisingly predictive of the environment score that the unit received in REF2021. Specifically, these topic proportions collectively explained 58.9% of the variance in environment GPAs, and 21.9% of the variance over and above the variance explained by the unit’s (standardised) FTE staff number, the (standardised) number of doctoral degrees it awarded, and its (standardised) grant income. In total, these metrics and the topic proportions from these eight topics collectively explained 69.1% of the variance in environment GPAs. Alongside these main findings, we also identified that environment statements that contained a lot of disciplinary-specific language tended to score higher than those which did not, although this effect was stronger for medical and biological disciplines, and weaker for the arts and humanities.

All the analyses we have reported in this paper are correlational in nature. Clearly, we were not able to experimentally manipulate the environment statements submitted to the REF and then assess the effect that these manipulations had on GPAs. Given this, care must be taken before assuming that the relationships we have reported are causal . Of particular concern is that some of our findings might be attributable to confounding factors. Indeed, in at least some cases this seems quite plausible. For instance, perhaps the reason why a research strategy focused on the REF seems to be negatively correlated with GPAs is that departments which have (relatively) low levels of research activity tend to both have a less mature research environment and also choose to write their environment statements using a higher proportion of REF terminology, as they have fewer pre-existing structures with pre-existing terminology to draw upon. We discuss this issue further below. Given this possibility of confounding factors, caution is required when interpreting our findings. Clearly, we cannot confidently draw causal conclusions in the absence of an experimental study (which would inevitably be of questionable external validity). Nevertheless, we can speculate.

REF environment scores are awarded through a process of human judgement. These are, by necessity given the volume of reading required of REF panellists, produced relatively rapidly. Many theories of human judgement emphasise how judgements are formed by comparing to-be-judged objects against prototypical instances sampled from memory (e.g. Fiedler 2000 , 2008 ; Stewart, Chater and Brown, 2006 ; Unkelbach, Fiedler and Freytag, 2007 ). Such theories would likely conceptualise reaching judgements about REF environment quality as a process which involves storing multiple exemplars of high- and low-quality statements in memory and, when encountering a new statement, generating a quality estimate by matching the features of the to-be-judged statements against those exemplars or prototypes ( Glöckner and Witteman 2010 ). This process need not be conscious, meaning that panellists are unlikely to be fully aware of the features they use to decide upon environment scores. Given this, it perhaps reasonable to suspect that if low-scoring environment statements typically have a given feature, then when panellists encounter a new statement with that feature, there may be a bias towards it receiving a lower score. In other words, if the majority of high-scoring environment statements that a panellist sees avoid using REF-heavy terminology, then in light of the decision-making literature on reasoning from prototypes and exemplars, it seems plausible that their judgements of future environment statements will be influenced by the presence or absence of this terminology, perhaps only unconsciously. If this account is correct, then the correlations between topic weightings and environment GPAs that we have reported above may well be, in part at least, causal.

How to write a ‘good’ environment statement

Assuming there remains a significant narrative element to REF people, culture and environment statements in 2029, what lessons might we learn from this analysis to support the crafting of written submissions that include all those features that are associated with high-scoring statements, and none of those features associated with low-scoring statements? We make eight recommendations.

First, we would avoid stating things that high-quality research environments would consider trivial. For example, we would not mention that most staff in our unit have doctorates, or that our staff attend academic conferences and write articles in academic journals. We would avoid using the phrase “research-active”, especially in an aspirational way, and we would not mention that our staff review articles for academic journals unless they had substantial editorial roles. In short, routine research activities should not be discussed in REF environment statements. Doing this is likely to give readers the impression that research is not a central feature of the unit’s work and reduce perceptions of the vitality of the unit’s research environment.

Second, when discussing research strategy, we would not give the impression that our research strategy is solely driven by the REF. Instead, our strategy would be organised around research centres, research groups, and departments. It would be focused on an academic discipline, not a “UoA”. The staff who led our submission would not be characterised as a “UoA Working Group”, and if we appointed other academic staff to REF leadership roles we would not mention it in our submission. Although we have robustly demonstrated that there is a relationship between conceptualising strategy using REF-centred terminology and receiving lower environment scores, it is less clear why this might be. One possibility is that the most research-intensive universities are sufficiently self-confident, and have a sufficiently long history of conducting research, to define their research activity in their own terms. In contrast, less research-intensive universities may need to create research infrastructure and strategies primarily in order to produce a respectable REF statement. If this were the case, we might expect the whole research enterprise in less research-intensive institutions to be more likely to be conceptualised in REF terms.

Third, we would not go into too much detail about the specific ways in which staff-related processes operate. For instance, we would not explain how decisions about sabbatical leave are informed by input from both the research committee and the teaching allocation committee. Similarly, details about which staff are involved at which stages in approving requests for conference travel funding would be omitted. Why might including detail of this sort be associated with lower GPAs? One plausible explanation is simply that including such details is a waste of space. As noted in the Introduction, REF environment statements are word limited, so including superfluous details might simply prevent the inclusion of content that would be causally associated with higher GPAs. This might be sufficient to generate a small negative relationship with GPAs (even though none would exist if there were no length restriction on submissions).

Fourth, we would not focus too much attention on how we support the career development of our ECRs. This finding is particularly surprising in light of the REF submission guidance’s statement that submissions should include “evidence of how individuals at the beginning of their research careers are being supported and integrated into the research culture of the submitting unit” (REF Panel Criteria and Working Methods 2019: 63). What might explain this apparent contradiction? An inspection of Figure 2 reveals that the negative relationship between discussing ECR development and GPA was driven by statements which included a relatively high proportion of this topic. Perhaps too much discussion of ECR development had the effect of crowding out space which could have been used for content that was more strongly associated with positive GPAs. Another possibility is that an excessive focus on ECR development might indicate to panellists that a unit feels that they have an unusually low proportion of senior established researchers in post.

Fifth, we would take care to make sure that we discussed career development and EDI, but not too much. The REF guidance emphasised that EDI should be discussed throughout submissions but, as shown in Figure 2 , some submissions clearly failed to follow this guidance, and these tended to receive low GPAs. However, some submissions seemed to discuss career development and EDI too much. Our analysis suggested that devoting ∼13% of the statement to this topic was optimal, with scores falling off for submissions with substantially higher or lower figures than this. Explaining why submissions which did not spend much time discussing career development and EDI tended to score poorly is straightforward: they failed to follow the clear instructions provided, and panellists may have concluded that they were poor places for minoritized colleagues to work. But why might have submissions which discussed these topics at length received lower scores? Again, one plausible account involves appealing to the length limitations of the environment statement template. Perhaps discussing career development and EDI was a qualifying criterion: not taking this issue sufficiently seriously would harm a submission, but once a submission successfully demonstrated that career development and EDI was a matter of concern, then further discussions on the topic became unnecessary. Instead, extra details on these matters had the effect of crowding out space that could have been productively used to discuss other issues associated with higher GPAs. A second possibility is that some statements mentioned EDI so often that it conveyed a ‘tick box’ approach rather than an authentic embedding. One final possibility is that an unusually high level of discussion of EDI issues might give the impression to reviewers that the unit felt that they had an unusually high number of issues in this area which required particular attention. Again, this might give the impression of a poor environment for minoritized colleagues.

Sixth, we would illustrate our research strategy by giving as many concrete examples as possible of how it has been implemented in practice. For example, if we provided pump-priming research funding to our staff, we would give an example of someone who had received funding, what they did with it, and what this led to. If we had particular strategies in place to facilitate interdisciplinary work, we would give an example of how this had led to a successful interdisciplinary workshop or funding application. If we had a policy on sharing research data, we might state the proportion of empirical papers in our submission where data had been shared online and give an example of how these datasets had been used by external colleagues in their own work. If we had a particularly generous study leave allowance for colleagues returning from parental leave, we might give an example of outputs or successful grant applications that had been produced as a result of this policy, and so on.

Seventh, we would mention our research funding and industrial partnerships as much as possible. This might seem superfluous, as panellists were provided with each unit’s grant expenditure alongside the written environment statement. However, our data suggest that mentioning funding and partnerships explained significant variance in GPAs over and above standardised grant income per FTE. 1 In sum, having high levels of grant income is insufficient: one must use it to provide evidence of a successful research strategy and environment as well as what the income enabled.

Eighth and finally, we would discuss our discipline as much as possible, particularly if we were writing a statement as part of a submission to a STEM UoA. For example, we would illustrate the success of our research strategy by discussing some of the important research findings it facilitated, we would name our research groups using well-understood disciplinary terms, and we would describe the work that our research funding allowed us to do, rather than merely state grant funding amounts. This finding that the use of discipline-specific language tends to be associated with higher environment GPAs is particularly interesting given the original suggestion of the Institution-Level Environment Panel Pilot that future REF exercises should abandon unit-level environment statements altogether ( REF 2022 ). Our finding that the scores produced by discipline-specific panellists were correlated with the amount of discipline-specific language submissions used, suggests that they may have used their domain expertise to come to decisions about submission quality. Clearly this would not be possible to the same extent, if at all, if environment were assessed at the institutional level by a panel made up of experts from a variety of disciplines—even if, as the REF (2022) pilot panel proposed, brief unit-level narratives were incorporated into the institutional-level statement. In sum, our results indicate that research environment assessed at the institutional level would likely be a different construct from research environment assessed at the unit level.

It has now been communicated that the next Research Excellence Framework will continue to assess an institution’s people, culture and environment at both institution- and discipline level, with the greater weight being given to the discipline-level assessment ( Joint UK HE Funding Bodies 2023 ). It is not yet clear the extent to which this element will look beyond the domains assessed in REF2021 (context, people, income & infrastructure, and collaboration & contribution), nor the extent to which the assessment will consist of quantitative indicators relative to narrative description. Regardless, our analysis may help institutions reflect upon what it means to have a high-quality research environment.

We are grateful to Hugues Lortie-Forgues, Victoria Simms, Steve Rothberg, and two anonymous reviewers for insightful comments on earlier drafts of this manuscript.

This work was partially supported by Research England, via an Expanding Excellence in England grant to the Centre for Mathematical Cognition, and the Economic and Social Research Council [grant number ES/W002914/1].

Conflict of interest statement. None declared.

Data associated with this manuscript are available at https://doi.org/10.17028/rd.lboro.23912499.v1 .

A regression predicting environment GPAs using only two independent variables, standardised grant income per head and the weightings associated with Topic 34 Industry Partners and Funding, could explain 36% of the variance in environment GPA. In this regression the standardised coefficients associated with the grant income metric and Topic 34 were β = 0.476 and β = 0.255 respectively, indicating that the amount of grant income a unit received was only slightly less than twice as important as how much it discussed it in its environment return.

Anthony L. ( 2022 ) AntConc (Version 4.2.0) [Computer Software] . Tokyo, Japan : Waseda University .

Google Scholar

Google Preview

Bence V. , Oppenheim C. ( 2005 ) ‘ The Evolution of the UK’s Research Assessment Exercise: publications, Performance and Perceptions ’, Journal of Educational Administration and History , 37 : 137 – 55 .

Blei D. M. , Ng A. Y. , Jordan M. I. ( 2003 ) ‘ Latent Dirichlet Allocation ’, The Journal of Machine Learning Research , 3 : 993 – 1022 .

Brassington L. ( 2022 ) Research Evaluation: Past, Present and Future . HEPI Report 152. Higher Education Policy Institute.

Brown R. , Carasso H. ( 2013 ) Everything for Sale? The Marketisation of UK Higher Education . Routledge .

Curry S. , Gadd E. , Wilsdon J. ( 2022 ) Harnessing the Metric Tide: indicators, infrastructures & priorities for UK responsible research assessment . < https://rori.figshare.com/articles/report/Harnessing_the_Metric_Tide/21701624/2/files/38515103.pdf > accessed 5 Feb 2024.

Derrick G. E. , Samuel G. N. ( 2016 ) ‘ The Evaluation Scale: Exploring Decisions about Societal Impact in Peer Review Panels ’, Minerva , 54 : 75 – 97 .

Fairclough N. ( 1995 ) Critical Discourse Analysis . London : Longman .

Fiedler K. ( 2000 ) ‘ Beware of Samples! A Cognitive-Ecological Sampling Approach to Judgement Biases ’, Psychological Review , 107 : 659 – 76 .

Fiedler K. ( 2008 ) ‘ The Ultimate Sampling Dilemma in Experience-Based Decision Making ’, Journal of Experimental Psychology: Learning, Memory, and Cognition , 34 : 186 – 203 .

French N. J. , Massy W. F. , Young K. ( 2001 ) ‘ Research Assessment in Hong Kong ’, Higher Education , 42 : 35 – 46 .

Geuna A. , Martin B. R. ( 2003 ) ‘ University Research Evaluation and Funding: An International Comparison ’, Minerva , 41 : 277 – 304 .

Gillies D. ( 2008 ) How Should Research Be Organised? London, UK : College Publications .

Glöckner A. , Witteman C. ( 2010 ) ‘ Beyond Dual-Process Models: A Categorisation of Processes Underlying Intuitive Judgement and Decision Making ’, Thinking & Reasoning , 16 : 1 – 25 .

Grimmer J. , Stewart B. M. ( 2013 ) ‘ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts ’, Political Analysis , 21 : 267 – 97 .

Jacobi C. , van Atteveldt W. , Welbers K. ( 2016 ) ‘ Quantitative analysis of large amounts of journalistic texts using topic modelling. ’, Digital Journalism , 4 : 89 – 106 .

Jensen E. A. , Wong P. , Reed M. S. ( 2022 ) ‘ How Research Data Deliver Non-Academic Impacts: A Secondary Analysis of UK Research Excellence Framework Impact Case Studies ’, Plos One , 17 : e0264914 .

Joint UK HE Funding Bodies . ( 2023 ) Research Excellence Framework 2028 Initial Decisions and issues for further consultation (REF/2028/23/01). < https://repository.jisc.ac.uk/9148/1/research-excellence-framework-2028-initial-decisions-report.pdf > accessed 5 Feb 2024.

Jones P. , Sizer J. ( 1990 ) ‘ The Universities Funding Council’s 1989 Research Selectivity Exercise ’, Beiträge Zur Hochschulforschung , 4 : 309 – 48 .

Kellard N. M. , Śliwa M. ( 2016 ) ‘ Business and Management Impact Assessment in Research Excellence Framework 2014: analysis and Reflection ’, British Journal of Management , 27 : 693 – 711 .

King’s College London and Digital Science ( 2015 ) The Nature, Scale and Beneficiaries of Research Impact: An Initial Analysis of Research Excellence Framework (REF) 2014 Impact Case Studies . Bristol, United Kingdom : HEFCE .

Manville C. , d’Angelo C. , Culora A. , Gloinson E.R. , Stevenson C. , Weinstein N. , Wilsdon J. , Haddock G. , Guthrie S. ( 2021 ) Understanding Perceptions of the Research Excellence Framework among UK Researchers . Cambridge, UK : RAND Corporation .

Manville C. , Guthrie S. , Henham M.-L. , Garrod B. , Sousa S. , Kirtley A. , Castle-Clarke S. , Ling T. ( 2015 ) Assessing Impact Submissions for REF 2014: An Evaluation . Cambridge, UK : RAND Corporation .

Marques M. , Powell J. J. , Zapp M. , Biesta G. ( 2017 ) ‘ How Does Research Evaluation Impact Educational Research? Exploring Intended and Unintended Consequences of Research Assessment in the United Kingdom, 1986–2014 ’, European Educational Research Journal , 16 : 820 – 42 .

Matthews A. , Kotzee B. ( 2022 ) ‘ Bundled or Unbundled? A Multi‐Text Corpus‐Assisted Discourse Analysis of the Relationship between Teaching and Research in UK Universities ’, British Educational Research Journal , 48 : 578 – 97 .

McCallum A. K. ( 2002 ) MALLET: MAchine Learning for LanguagE Toolkit . < http://mallet.cs.umass.edu > accessed 5 Feb 2024.

Mellors-Bourne R. , Metcalfe J. , Gill A. ( 2017 ) Exploring Equality and Diversity using REF2014 Environment Statements , CRAC LTD Report to HEFCE.

Pardo-Guerra J. P. ( 2022 ) The Quantified Scholar: How Research Evaluations Transformed the British Social Sciences . Columbia University Press .

Pinar M. , Horne T. J. ( 2022 ) ‘ Assessing Research Excellence: evaluating the Research Excellence Framework ’, Research Evaluation , 31 : 173 – 87 .

Poppler ( 2022 ) [Computer Software] < https://poppler.freedesktop.org > accessed 5 Feb 2024.

REF ( 2019 ) REF Panel Criteria and Working Methods . < https://archive.ref.ac.uk/media/1450/ref-2019_02-panel-criteria-and-working-methods.pdf > accessed 5 Feb 2024.

REF ( 2022 ) REF Institutional-level Environment Pilot: Report of the Pilot Panel . < https://archive.ref.ac.uk/media/1908/ref-2021-ilepp-final-report.pdf > accessed 5 Feb 2024.

Reichard B. , Reed M. S. , Chubb J. , Hall G. , Jowett L. , Peart A. , Whittle A. ( 2020 ) ‘ Writing Impact Case Studies: A Comparative Study of High-Scoring and Low-Scoring Case Studies from REF2014 ’, Palgrave Communications , 6 : 1 – 17 .

Sivertsen G. ( 2017 ) ‘ Unique, but Still Best Practice? The Research Excellence Framework (REF) from an International Perspective ’, Palgrave Communications , 3 : 17078 .

Stewart N. , Chater N. , Brown G. ( 2006 ) ‘ Decision by Sampling ’, Cognitive Psychology , 53 : 1 – 26 .

Sutton E. ( 2020 ) ‘ The Increasing Significance of Impact within the Research Excellence Framework (REF) ’, Radiography , 26 : S17 – S19 .

Swedish Government . ( 2016 ) Regeringens Proposition 2016/17:50 . Kunskap i samverkan—för samhällets utmaningar och stärkt konkurrenskraft .

Terämä E. , Smallman M. , Lock S. J. , Johnson C. , Austwick M. Z. ( 2016 ) ‘ Beyond Academia–Interrogating Research Impact in the Research Excellence Framework ’, Plos One , 11 : e0168533 .

Thomas D. A. , Nedeva M. , Tirado M. M. , Jacob M. ( 2020 ) ‘ Changing Research on Research Evaluation: A Critical Literature Review to Revisit the Agenda ’, Research Evaluation , 29 : 275 – 88 .

Thorpe A. , Craig R. , Tourish D. , Hadikin G. , Batistic S. ( 2018a ) ‘ Environment’ Submissions in the Uk's Research Excellence Framework 2014 ’, British Journal of Management , 29 : 571 – 87 .

Thorpe A. , Craig R. , Hadikin G. , Batistic S. ( 2018b ) ‘ Semantic Tone of Research ‘Environment’submissions in the UK’s Research Evaluation Framework 2014 ’, Research Evaluation , 27 : 53 – 62 .

Unkelbach C. , Fiedler K. , Freytag P. ( 2007 ) ‘ Information Repetition in Evaluative Judgements: Easy to Monitor, Hard to Control ’, Organizational Behavior and Human Decision Processes , 103 : 37 – 52 .

Watermeyer R. , Derrick G. E. ( 2022 ) ‘ Affective Auditing: The Emotional Weight of the Research Excellence Framework on Middle Management ’, Research Evaluation , 31 : [rvac041]. 498 – 506 . https://doi.org/10.1093/reseval/rvac041

Email alerts

Citing articles via.

• Recommend to your Library

Affiliations

• Online ISSN 1471-5449
• Print ISSN 0958-2029
• Copyright © 2024 Oxford University Press
• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

Science and Technology in the Academic Enterprise: Status, Trends, and Issues (1989)

Chapter: the research environment.

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

EMERGING TRENDS 17 Emerging Trends The ability of universities to broaden their missions and play a larger role in the nation's research enterprise will depend on the resolution of three sources of tension, each pulling at the fabric of the enterprise. The first strain on the enterprise is slow adaptation to an increasingly complex research and educational environment; the organization, culture, and resources of academic institutions and their research sponsors constrain their response to new demands and opportunities. The second source of stress on the enterprise is the replacement of retiring high-quality research personnel during the next decade; it may not be possible, given the current production level of research scientists and engineers. The third source emanates from the need to sustain the quality of current research institutions and programs, which is increasingly expensive to do and—in an era of severely constrained fiscal resources—increasingly difficult. THE RESEARCH ENVIRONMENT The environment in which the academic research community must function will increase in complexity. National and international economic, political, and social cross-currents influence the priorities, topics, and contexts of scientific investigation. These influences are combining to challenge the traditional way scholars and their host institutions operate and relate to each other. Furthermore, many new scientific and technological opportunities require more flexible, cross- disciplinary relationships both within and among universities, industries, and governments. There are many factors at work here. First, important and exciting advances in fundamental science are occurring are creating more complex questions on the research frontier and many of the questions are more frequently in multi-disciplinary settings at the interface between disciplines. Furthermore, some traditional fields, such as molecular biology and microelectronics, are merging with other fields or being redefined. Second, as product life cycles become shorter, advances in fundamental knowledge become more relevant to technology development. As a result, industries, universities, and financial institutions are developing sophisticated relationships that include a multiplicity of formal and informal structures. Some faculty members, for example, are assuming entrepreneurial roles, including developing relationships with non-academic organizations to pursue the commercial development of their research. Third, international cooperation is intensifying in many scientific and engineering fields. The growing research capabilities of other nations provide new opportunities for collaboration—especially in astronomy, oceanography, and high- energy physics—that require large capital investments. International cooperation is also required for research on such problems as global climate change, ozone depletion, and acid rain. New technologies increasingly shape the scholarly agenda in the sciences and engineering. State-of-the-art instrumentation allows for experiments requiring heretofore un-achievable precision and scale. New generations of computers make possible large-scale 

EMERGING TRENDS 18 data analysis and provide the mechanism for rapidly transferring and sharing information among institutions, organizations, and nations. News of new processes and products of scientific research reach an ever-wider U.S. audience. To the extent that popularization contributes to public understanding of science, it enhances political support. But it also brings greater societal scrutiny to the research enterprise. There is, for example, growing public pressure on federal regulatory and grant-making agencies to control the use of toxic substances and radioisotopes, and experiments involving animals. In addition, societal intervention in the research agenda is increasingly exercised through the courts, notably in environmental protection, radiation and carcinogen disposal, and the release of genetically engineered material. In addition to increasing regulatory complexity in some fields, the lack of regulations in other fields is also a problem—often forcing researchers to curtail or abandon lines of inquiry in areas such as biotechnology. The most pronounced recent trend is state and local regulation of research. A few state, county, and city governments have begun to influence the conduct of local university research through controls on the type and location of university facilities and on research protocols, such as the use and care of test animals and the use of genetically altered organisms. Should this trend become more widespread, investigators and their host institutions would have to adapt to a changing array of costly reporting requirements, safeguards, controls, and regulatory supervision. Universities and research sponsors face difficulty in rapidly adapting to a changing research environment. In response to the changing research environment, some members of the academic enterprise are testing innovative strategies for organizing, conducting, managing, and financing research. Rapid adaptation to new demands and opportunities in the research area, however, is slowed by many factors—including tradition, inertia, the competition for university resources, the demands of the university's educational mission, and the aging of faculty—impinging on the current organization, culture, and resources of university-based scholars and their funding agencies. There is growing debate within universities over the ability of the current disciplinary and governance structures to respond adequately to the expanding research agenda, as well as to find an appropriate balance of commitments to scholarship, education, and public service. New research opportunities often require more flexible budgeting and assignment of research faculty, inter-disciplinary approaches, expansion of non-faculty research personnel, extra-departmental initiatives, and allowance for faculty entrepreneurial activity. Furthermore, larger-scale multi-disciplinary research efforts require hierarchical management and more centralized governance structures for rapidly making strategic decisions and for inter- departmental planning. In addition, the intense regulatory environment in many areas of research requires active participation by the institution's administration in deciding faculty research topics and protocols, as well as in serving as a necessary buffer against unwarranted outside interference. On the other hand, the present university disciplinary structure has proved adaptable to new research opportunities and, more importantly, provides a necessary, albeit cumbersome, system for quality control through peer review. Young faculty, who are 

The U.S. academic research enterprise is entering a new era characterized by remarkable opportunities and increased strain. This two-part volume integrates the experiential knowledge of group members with quantitative data analyses in order to examine the status of scientific and technological research in academic settings. Part One reviews the status of the current research enterprise, emerging trends affecting it, and issues central to its future. Part Two is an overview of the enterprise and describes long-term trends in financial and human resources. This new book will be useful in stimulating policy discussions—especially among individuals and organizations that fund or perform academic research.

READ FREE ONLINE

Welcome to OpenBook!

You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

Do you want to take a quick tour of the OpenBook's features?

Show this book's table of contents , where you can jump to any chapter by name.

...or use these buttons to go back to the previous chapter or skip to the next one.

Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

To search the entire text of this book, type in your search term here and press Enter .

Share a link to this book page on your preferred social network or via email.

View our suggested citation for this chapter.

Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

Get Email Updates

Do you enjoy reading reports from the Academies online for free ? Sign up for email notifications and we'll let you know about new publications in your areas of interest when they're released.

Sabine Müller

On creating a good research environment

31 March 2021 | doi:10.5281/zenodo.463628 | 1 Comment

Sabine Müller on the hierarchical system of German academia and why it could be a problem for the wellbeing of young academics and Ph.D. candidates. She compares it to her experiences at Oxford University and sheds light on the differences between the two research cultures.

“Researchers say that their working culture is best when it is collaborative, inclusive, supportive and creative, when researchers are given time to focus on their research priorities, when leadership is transparent and open, and when individuals have a sense of safety and security. But too often research culture is not at its best.” [“What researchers think about the culture they work in”, Wellcome Trust and Shift Learning (London 2020), p. 3 (subsequently referred to as “What researchers think”)] The executive summary of the 2020 Wellcome Trust study on research culture goes on to describe how “many [researchers] are often missing out on critical aspects of good management … [a]nd worse, many have experienced exploitation, discrimination, harassment and bullying.” Notably, members of minority groups more often experience the latter. These results echo those of previous surveys, such as those conducted by Advance HE or the journal Nature in the UK – and which illustrate that these issues are on the radar of public debate. (Woolston, 2019) The situation in Germany is hardly any different, though of course data in Germany is scarce and hardly sufficient to make any reliable statements about early career researchers’ emotional situation. (Most notably, the German Centre for High Education Research and Science Studies (DZHW) runs a National Academics Panel Study since 2017 which promises to give further data on the condition and well-being of doctoral researchers.) Notably, existing studies are frequently a reaction to incidents that reached public attention (Albott, 2019), or rely on surveys by the concerned group such as by the networks of doctoral researchers of non-university research organisations. Accordingly, the existence of guidelines for managing power abuse or mental health are confined to institutions which have struggled with cases. Otherwise the silence on the topic by reputable research organisations such as the DFG or the academies is overwhelming . The reasons for this gap may be manifold. German academia is not immune to the complex range of problems such as non-transparent leadership, a lack of inclusiveness, harassment or mental health issues which resist a positive research culture. Each of these issues by themselves has numerous causes, but all of them are amplified by a system which lays “an excessive focus on measuring performance” as well as institutional structures such as the accumulation of responsibilities and decision making as well as steep hierarchies. (Shore & Wright, 1999) It is the latter aspects which I would like to focus on as crucial elements required in order to foster a cultural change for a positive research environment, and which, compared to the bigger systemic issues could be rather easily fixed.

As a career development adviser in Germany, I was frequently confronted with the following “argument” during discussions with senior researchers about the working conditions of early career researchers: “It was like this when I did my PhD, so why should that not work nowadays?” I always wondered what exactly this was meant to say? Unpacking this claim to myself it seems to implicitly suggest 1) the person, too, did not attain their PhD in good conditions, 2) but the fact that they succeeded makes them think that it was not so bad after all. Is the rationale behind this that a “rough school” toughens people up to prepare them for academic life? In fact, I often also heard that doctoral researchers today are not only too sensitive but too demanding. But is the consequence that only people who are willing to toughen up stay in academia? Besides the questionable psychological rationale, I wonder whether we really think that this is what academia needs: tough personalities? Putting aside the universe of unconscious biases which is touched by such a question, shouldn’t academic work and life not be guided – even more than any other branches of the labour market – by the principle of reason, multiperspectivity, openness, integrity and such, rather than of the dull workings of unconscious bias and self-perpetuation? And should the system not aim to do everything for those qualities to be able to unfold and thrive? Responding to such a statement from my own personal experience, I often felt awkward since I did not share this experience. And this difference in the culture of dealing with issues such as discrimination, mental health, diversity and welfare, as well as power abuse has struck me most notably upon my return to Germany after I had spent eight years at the University of Oxford taking up a position in career development in a research organisation. My experience during my doctoral and postdoctoral research was a very positive one – in many respects, I had the time of my life – and I felt that encouraging people to create an environment for such a good experience would be crucial. In the following piece, I focus on some landmarks of this positive (!) experience in my academic career in order to point to how a fundamental change of culture needs to be human centered, and attend to individual experience. 

I was granted a first memorable insight into a different mind-set before I left for my one year Master’s, which led to my doctoral research at the University of Oxford. I was confronted with the choice between 32 colleges. I was inclined to apply at the college to which my future supervisor was affiliated – a thought which very much agreed with the logic of the German system I was socialized in where doctoral researchers are frequently not only naturally affiliated with the “Lehrstuhl” of their supervisors but also seem to enter into a sort of patronship relation expressed in the still prevalent German term “Doktorvater/-mutter”. However, my supervisor asked me to consider that in case of disagreement or conflict, it would be advantageous to be able to have an independent college adviser to turn to. The sense of responsibility expressed in this thoughtfulness with respect to providing an environment to my advantage profoundly shaped my own actions along the subsequent years as doctoral and postdoctoral researcher, as senior subject tutor and lecturer at Oxford University and beyond. Supervision training might help, but can only partly address the care at work here. The ambition to promote your doctoral researcher, so that s/he can realise her/his potential is connected to a sense of duty to pay attention to the welfare of your supervisee. This attention is promoted and aided by structural aspects. Beside my supervisor – who I am lucky to say was a most conscious and inspiring researcher with whom I met every other week, and who conscientiously read every essay, chapter or anything else I ever submitted – I was then assigned a college adviser. In my case this person was an éminence grise in my field who, as tradition would have it, invited me to a talk at the fireplace and imparted his wisdom to me – and, last but not least, a faculty adviser who was there to offer further opportunities to talk about the programme of my thesis and to whom both my supervisor and I had to submit a progress report by the end of each term. Moreover, the degree at Oxford has a clear milestone system in which supervision and assessment are separated from each other: the vivas for the transfer of status as “Probationer Research Student” to DPhil Candidate after one year into your degree as well as the confirmation of status after two years is taken by two faculty members. The assessment of the submitted final dissertation lies in the hands of an internal and external examiner. This way of organization ensures that the role of the supervisor is focused to act as adviser and to support their supervisee as best as they can. Of course, this means some control for the supervision process: Failure to bring your supervisee to successfully finish their degree will not have consequences for any academic but is not as easily obscured by the possibility to drag the doctoral research on or by marking the thesis accordingly. At the same time, the shared roles opened the opportunity for me, as the supervisee to connect and frankly discuss with other senior academics who took my work seriously. 

Thus, the transparency of a clear milestone system, which details what is expected from the student as well as the separation of the roles of supervision, monitoring and assessment, has the potential to minimise the risk of power abuse and lifts the weight from the relation between supervisors from the start. It affords the supervisee the opportunity to discuss her or his work throughout the process with various researchers, to gain more perspectives and develop an independence of thought and a network from the get-go. Combined with the opportunity to frequently share your intellectual thoughts with established experts in your field and beyond made doctoral and postdoctoral research particularly worthwhile.

I would like to add that as a senior subject tutor for German Studies I experienced the advantages of this disentanglement of examination and supervision for myself: the faculty assigns a committee which designs the end of year exams. Marking and assessment were organised anonymously in an annual rotating system of examiners. Both procedures entail multiple advantages: not only do they limit the power of tutor or supervisor but they also relieve both from that burden of power. Not being the examiner, you can truly fulfil the role as adviser, coach and teacher and accompany your students along their development. Reaching out to your tutor or supervisor is easier, if you do not have to fear any repercussion on your performance. In this context, I also learned to appreciate the carefully built college and university community which provided a network to support students and lecturers alike. It ranges from the so-called common rooms with their mentor for freshers and trained peer advisers to college and university counsellors as well support staff for people of various religious and ethnical background on campus. Coming from a German university, this amount of attention and care which unloaded over my head was at first rather overwhelming and I confess I thought it unnecessary. But over the years, I learned to appreciate this culture which aspired to keep people well and enable them to enjoy their time at the university. Especially later, as a senior subject tutor, when my contract stated in no uncertain terms that tasks comprised the welfare of my students, this community recognised the limits of my competencies and acknowledged the need for welfare offers. 

Another major landmark remains the handling of admission and application procedures. Perhaps it is worth explaining that student admissions at Oxford is a highly professional and formal process which stretches over two weeks in December after the autumn term. Not only are we dealing with standardised applications which aim to highlight the potential of each candidate. Each applicant invited to interviews has the right to get at least two interviews with different academics to assess their performance. In fact, it is a very intricate system with the objective to select students with high potential, no matter their background. In my first year, I was asked to write the protocol for admission interviews and even for that rather small task, I had to complete an online course on legal liabilities, correct interview methods, harassment, discrimination and the mechanism of unconscious bias. As senior subject tutor responsible for admissions in your subject area, I had to take another, more extensive course with on- and offline elements. These courses were a necessary eye-opener to topics which had never been addressed, even in the student council of my German university. It set my expectations of what I consider to be a professional application procedure and to this day I find it hard to accept that none of this type of elementary interview training, which raises awareness of everyone’s unconscious blind spots concerning bias and awareness, is a required standard at German universities or research organisations. It would be easy to implement part of a structured onboarding for each and every academic at the university and at least make the recruiting procedure a bit fairer. 

To sum up, I would like to make clear that I am aware that problems prevail in the UK, as the quoted Wellcome Trust study illustrates. I also want to point out that reason why welfare at Oxford and Cambridge is paid such attention is not entirely altruistic: for a long time, these Universities had to deal with the reproach of higher suicide rates – a critique which cannot be sustained (Hawton et al., 2012).  In addition, it is often pointed out that these institutions are only accessible to elites, which, at an undergraduate level, is very true. At the same time, Oxbridge institutions  understand that in order to attract the best academics they have to cater to people’s wellbeing as human beings in every aspect. So strategic deliberations and monetary concerns are certainly central drivers for the implementation. However, this does not devalue the learnings from such an experience: a collaborative, open, transparent and overall friendly environment relies on the mind-set of the academics who acknowledge the responsibility for their supervisees. This mind-set is supported by structures that foster transparency, independence and exchange by clearly laying out the demands and milestones of a doctoral course (without the need to make people go back to school) by separating the roles of supervision, monitoring and assessment, by carefully building a community with low-threshold support structures catering to various backgrounds as well as training to raise awareness to biases, harassment, stress symptoms etc. None of these suggestions are new but maybe not enough people have experienced how powerful they can be in their small workings and, thus, not enough people can or want to pass on this kind of experience.

Albott, Alison: Germany’s prestigious Max Planck Society investigates new allegations of abuse, in Nature (online) (9 July 2019) https://www.nature.com/articles/d41586-018-05668-y / doi: https://doi.org/10.1038/d41586-018-05668-y

Hawton K., Bergen H et.al : University Students over a 30-years period, in Social Psychiatry and Psychiatric Epidemology 47 (2012), p.43-51, https://doi.org/10.1007/s00127-010-0310-3 (a summary is provided: https://www.psych.ox.ac.uk/publications/168323 ). Further information can be obtained at the Office for National statistics :

Shore, C., & Wright, S. (1999). Audit Culture and Anthropology: Neo-Liberalism in British Higher Education. The Journal of the Royal Anthropological Institute, 5(4), 557-575. doi: 10.2307/2661148 See also:  “A cry for help”, in Nature 575 (14 November 2019), p.257-258: https://www.nature.com/articles/d41586-019-03489-1

Woolston, Cristof: PhDs: torturous truths, in Nature 575 (13 November 2019), p.403-406 https://www.nature.com/articles/d41586-019-03459-7

Thanks for this, Sabine. You very well present the PhD scholar’s perspective. From a professor’s or supervisor’s perspective it is in my view even worse. I remember one of my first PhD examinations in Germany – not as a supervisor. I thought the PhD was really poor and wanted to be nice and make it pass but give it a very bad mark. I talk to a colleagues about this and she adviced me “Are you crazy, you cannot do that!! His supervisor will interpret this as an open war”. I learnt that the assessment of a PhD candidate is also assessment of the supervisor, so if you want to punish a colleague who has not supported you in another situation, you can do it by marking his candidate poorly. The other way around as well, of course: you’ll give high marks to the the PhD candidates of your friends. You can imagine yourself what kind of informal “economy” that goes on among supervisors/professors. I have been part of many PhD examinations, and I tell you that there is no system at all of good candidates getting good marks and bad candidates getting bad marks. What is negotiated at a PhD examination is very much the standing of the professors and their interrelations. A feudal system.

Submit a Comment Cancel reply

Your email address will not be published. Required fields are marked *

First we need to confirm, that you're human. eighty four − seventy five =

Author info

Sabine Müller is currently Adviser for Education and Participation in the Digital World at Wikimedia Deutschland. Before, she was a research consultant for humanities and educational research as well as career development at the head office of the Leibniz Association. She holds a DPhil from Oxford University where she also worked as senior subject tutor for German Studies at St John’s and Magdalen College as well as affiliate postdoc on embodied cognition and narration.

Academia.edu no longer supports Internet Explorer.

To browse Academia.edu and the wider internet faster and more securely, please take a few seconds to  upgrade your browser .

Enter the email address you signed up with and we'll email you a reset link.

• We're Hiring!
• Help Center

Understanding the Research Environment: Identifying Areas for Future Work

Science and technology policymaking has been an area of importance for developed and developing countries alike. However, it is often observed that there is a lack of literature and evidence on the way research has been conducted especially the environment in which the knowledge production happens. In the current review we have attempted to bring together the different issues that pertains to research environment from a global and developing country perspective. Specifically we have explicated the gaps where further studies and knowledge needs to be generated to have an understanding of the factors which impacted the research per se.

Related Papers

Knowledge, Technology & Policy

Wesley Shrum

International Journal of Business and Economics Research

Yassir Hussain

Helena Barnard , Robin Cowan

Do world-leading researchers from developing countries contribute to upgrading locally, or do they disengage from the local context? The paper investigates the scientific collaborations of university-based science and technology researchers in the database of the South African National Research Foundation (NRF), and analyses the co-authorships of researchers who were ranked by the NRF during the 2001–2007 period. To establish the extent to which a researcher can access knowledge outside the South African academic science and technology research community, and share it inside that community, we develop a measure of ‘gatekeeping’. The evidence suggests that there is not a local/global trade-off in knowledge creation in academia in the developing world, and that the world-leading researchers in developing countries may play an especially important role as conduits of new knowledge in their country.

Rituparna Bhattacharyya

One of the most significant current discussions is about building research capacity in resource limited countries. In India, the issue has grown its importance in the light of Prime Minister, Dr Manmohan Singh&#39;s growing concern on the fact that India&#39;s relative position in the world of science and technology has been declining and been overtaken by countries like China. Although, India&#39;s Research & Development (henceforth, R&D) expenditure has risen from close to Rs. 162,000 million in 2000-01 to nearly Rs. 378,000 million seven years later, however, according to statistics published by Department of Science and Technology, Government of India, it is less than 0.9% of the Gross Domestic Product (GDP) which is expected to rise to only 2% by the end of the 12th Plan period (2012-2016). India ranks 10th in terms of published citable scientific documents (1996-2010) but the quality (H index) of the research output ranks far behind and stands at 24th position. Importantly, in...

... Research in science and technology in the developing world Helena Barnard, Robin Cowan, Moritz Müller United Nations University – Maastricht Economic and social Research and training centre on Innovation and Technology Keizer Karelplein 19, 6211 TC Maastricht, The ...

Subbiah Arunachalam

There is a vast difference between the rich and poor countries in every respect. The difference is very pronounced in scientific and technical research, in terms of both volume and impact. Indeed the distribution of science is even more skewed than the distribution of ...

Desde el Herbario CICY - Herbario CICY

Akansha Srivastava

Studies have revealed that collaborative Research and Development (R&D) activities are difficult to manage and control since they involve a complex process, this paper investigates the core factors that affect collaborative R&D and identifies a value added model for the economic growth of developed and developing nations. Backing from the government and Foreign Direct Investment (FDI) policies should be encouraged to attract global R&D investment and collaboration. Role of Multi-National Enterprises (MNE’s) in Asia’s emerging economies such as China, India and Russia etc. are elaborated to further support that the management of R&D is essential for building a stable relationship. Conclusion of the paper reveals that as the developed economies enjoy the benefits of access to large market and diversification of products, the developing economies on the other hand get higher R&D investments and access to abundant resources. Developing economies learn a lot from developed economies in t...

Loading Preview

Sorry, preview is currently unavailable. You can download the paper by clicking the button above.

RELATED PAPERS

Caroline Wagner

Science and technology policy for development: …

Lea Maria L S Velho

Blucher Engineering Proceedings

Thiago Caliari

agriserviceethiopia.org

Tesfaye Beshah

N.V. Varghese Varghese

American Journal of Public Health

Walter Mendoza

Space and Culture, India

Ewelina Niemczyk

International Journal of Education …

Brij Mohan Gupta

Dr Isa Abdulkadir

Donizeti Souza

Evanthia Schmidt

Razia Sultana

The International Journal of Human Resource Management

Fabio Zicker

South African Journal of Science

Bernard Slippers

Philippe Mustar

Wajih Abdallah

Anthony egeru

Domenico De Martinis

International Journal of African and Asian Studies

Tony Nwanji

PLoS Biology

Milena Holmgren

Anna-Katharina Hornidge, Prof. Dr.

•   We're Hiring!
•   Help Center
• Find new research papers in:
• Health Sciences
• Earth Sciences
• Cognitive Science
• Mathematics
• Computer Science
• Academia ©2024

Research environment

ORI  Introduction  to RCR: Chapter 7. Mentor and Trainee Responsibilities

University of Michigan Mentoring Guidelines

Email Updates

• Principal Investigator

Principal Investigator (PI)

• OVERALL ROLE/RESPONSIBILITY
• ADMINISTRATION
• PROPOSALS & AWARDS
•  Pre-Award
•  Post-Award

Material Transfer Agreements (MTA)

•  Closeout
•  Effort Reporting
• RESEARCH CONDUCT
•  Human Research Participants

Clinical Trials

Animal subjects.

•  Research Safety
•  Research Integrity
•  Conflict of Interest
•  Patents and Inventions
•  Export Controls

Research Admin Units

• Center for Comparative Medicine (CCM)
• Institutional Animal Care and Use Committee Office (IACUC)
• Innovation and New Ventures Office (INVO)
• Institutional Review Board Office (IRB)

Export Controls & International Compliance

• Research Administrative Offices
• Research Communications
• Research Development (ORD)
• Research Financial Administration

Research Integrity

• Research IT

Research Safety

• Sponsored Research
• University Research Institutes and Center Administration (URICA)

Overall Role/Responsibility

Direct and oversee all research activities and foster a culture of research integrity. Responsible for fiscal and administrative management of research. Conduct research in an objective and unbiased manner in compliance with policies and regulations. While the PI may delegate responsibility for some project activity to others, the PI is ultimately responsible for compliance with all applicable regulations and policies and for ensuring a safe research environment, meaning one that is inclusive and free from any form of discrimination or harassment.

Northwestern University researchers can use this chart to pre-determine Principal Investigator eligibility to serve on an  IRB  protocol,  IACUC  protocol or a  Sponsored Research  proposal/award.

Administration

• Manage research staff, including co-investigators, post-doctoral trainees, fellows, students, technicians and lab managers
• Oversee the training and mentoring of post-doctoral trainees, fellows, and students
• Assure that all key research personnel have met training requirements
• Coordinate with school, department, and central administration to ensure that sponsored research activities are in accordance with all applicable regulations, policies, and procedures
• Ensure appropriate resources for research conduct
• Review, prepare, and submit results for publication and register publication, as required by sponsor

Proposals and Awards

Proposal submission.

• Review funding opportunity announcements
• Prepare technical and/or scientific proposal
• Develop and coordinate budgets, administrative elements, and materials, including materials for subcontracts
• Coordinate large proposals with the Office of Research Development (ORD)
• Understand and comply with institutional limited submission process
• Complete proposal application and submissions by required deadlines

Pre-Award Set-Up

• Ensure pre-spending (at-risk/advance) accounts are requested from Sponsored Research
• Complete “Just In Time” documentation requests from sponsor
• Provide estimate of sponsored project effort percentage
• Modify project scope and coordinate all administrative items, including budget, as necessary
• Direct all technical and administrative activities
• Review and confirm notice of award and award setup, including budget reconciliation
• Request issuance of subcontracts
• Review and approve any revisions to project scope, budget or changes in effort or other activities that may require prior approval from sponsor
• Monitor expenditures regularly to ensure that funds are managed in compliance with sponsor terms and conditions and only expended to directly support and benefit the project
• Ensure the accurate and timely submission of all required reports throughout the life of the award
• Confirm that all award records and data are maintained accurately and consistently over the award life cycle and through the sponsor and University’s records retention period
• Monitor subrecipient and consultant activity, including review and approval of invoices
• Responsible for deficits and disallowances incurred against an award
• Oversee confirmation and monitoring of program income
• Coordinate with the Accounting Services for Research and Sponsored Programs (ASRSP) office for all financial matters including fiscal compliance, cash application, financial audits, and effort reporting
• Communicate plans to transfer a grant to the sponsor, Sponsored Research, and other institutions
• Initiate a request to Sponsored Research for an MTA prior to any materials being sent to collaborators
• Review and approve all project expenses to ensure completion and compliance, including financial reports, final invoices, program income, and subrecipient performance
• Oversee completion of closeout documentation and ensure that all reporting requirements are met
• Confirm that all periodic and final technical and invention reports are submitted
• Work with ASRSP to resolve sponsored funds collection issues
• Certify budget statements at the end of the award

Effort Reporting

• Complete effort commitment profile
• Complete salary planning and distribution
• Disclose existence of appointments at other institutions to departmental chair
• Disclose and notify chair and dean when proposing cost share for proposal submission
• Fulfill and manage effort commitments of project members
• Certify effort by the required deadline
• Resolve effort discrepancies
• Ensure appropriate training of project staff

Research Conduct

Human research participants.

• Ensure the protection of the rights and welfare of human research participants
• Oversee submission to, and approval by, the Institutional Review Board (IRB) and confirm protocol/grant congruency
• Ensure the IRB approves the protocol prior to initiation
• Conduct all human participant research according to the approved protocol, relevant regulations, laws, institutional policies, and the Investigator Manual (HRP-103)
• Obtain IRB approval prior to the initiation of modifications to a study, except for those changes that are implemented to eliminate immediate hazards to participants
• Respond to all compliance issues, including participating in post-approval monitoring
• Confirm and monitor that the research team is trained and in compliance with IRB policies and procedures, federal and state regulations, and other relevant University policies
• Promptly suspend or terminate research, as appropriate, and submit reports of these actions and other reportable events to the IRB via Reportable New Information (RNIs) submissions
• Ensure clinical trials are conducted in accordance with Good Clinical Practice (GCP)
• Work with staff to prepare and submit case report forms for clinical trials
• Follow required closeout procedures (see ASRSP)
• Maintain current and accurate records in the Clinicaltrials.gov database
• Oversee animal study protocol submission to, and approval by, the Institutional Animal Care and Use Committee (IACUC) and provide a full copy of grant for protocol/grant congruency
• Ensure that the animal study protocol is IACUC approved prior to implementing animal work and that all aspects of animal care and use are conducted according to the approved protocol
• Ensure each grant has a separate approved animal study protocol
• Review/confirm animal charges and reconcile with the Center for Comparative Medicine (CCM)
• Confirm and monitor that research team is trained and qualified for the procedures they will perform, including compliance with all Occupational Health and Safety enrollment requirements
• Comply with all federal, state and local regulatory and institutional inspections and compliance investigations
• Inform the IACUC Office when custom antibodies or tissues are obtained from a third party
• Notify the IACUC Office of external collaborations involving animal work
• Notify the IACUC Office of animal work supported by funds from another institution
• Participate in Post-Approval Monitoring (PAM) visits
• Model, demonstrate and enforce safety and health compliance
• Mentor trainees and lab staff in relation to safety issues
• Register lab with Research Safety and keep research profile current
• Ensure consistent use of proper protective clothing and usage of materials
• Report any incidents that occurred during the performance of laboratory work activities that resulted in or could have led to injury or damage to property.
• Ensure that all lab members leaving the lab have cleaned up their assigned spaces before they depart
• Conduct research in an ethical and compliant manner
• Ensure integrity of all research activities and data
• Uphold secure and ethical data use, data confidentiality, and compliant data management, sharing, ownership, and retention practices
• Supervise and ensure the integrity of the research of all trainees and research staff
• Ensure Responsible and Ethical Conduct of Research (RECR) training for NSF supported undergraduate students, graduate students, and post-doctoral fellows as well as all trainees, participants, and scholars supported through applicable NIH awards
• Implement and follow responsible authorship practices, including adherence to discipline-specific guidelines

Conflict of Interest (COI)

• Meet initial and ongoing Conflict of Interest (COI) training and disclosure requirements
• Identify which individuals on the study team meet the definition of "investigator" and are subject to the COI requirements
• Adhere to the requirements of COI management plans
• Adhere to COI policies and processes and to all federal, state, and local regulations and sponsoring agency policies and procedures

Patents and Inventions

• Promptly disclose all inventions to the Innovation and New Ventures Office (INVO) using the on-line Inventor Portal must be completed at least 3-4 weeks prior to public disclosure)
• Provide accurate sources of funding on the Invention Disclosure form
• Communicate timely information to INVO, such as chartstring and/or account information, confidentiality agreements, technology development updates, etc.
• Adhere to Northwestern’s Export Control Compliance Policy and federal policies
• Adhere to federal regulations related to malign foreign talent programs
• Review and follow improper foreign interference guidelines
• Report potential research security issues to the Export Controls & International Compliance team

Warning: The NCBI web site requires JavaScript to function. more...

An official website of the United States government

The .gov means it's official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• Browse Titles

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Dusetzina SB, Tyree S, Meyer AM, et al. Linking Data for Health Services Research: A Framework and Instructional Guide [Internet]. Rockville (MD): Agency for Healthcare Research and Quality (US); 2014 Sep.

Linking Data for Health Services Research: A Framework and Instructional Guide [Internet].

• Hardcopy Version at Agency for Healthcare Research and Quality

2 Research Environment

A foundational element of any research project is the research program environment. In the context of comparative effectiveness research (CER) using linked data, a secure and well-performing environment is important for several reasons, including that it helps build and assure trust between researchers and the providers of sensitive data, be they patients, registry administrators, insurance claim administrators, or others. If data providers are confident that a research partner has strong administrative and technical security systems and takes data security seriously at a programmatic level, they will be more confident in providing sensitive data to the researchers, including data with unique identifiers. As we describe in Chapters 4 and 5 of this report, linkage quality is typically much stronger when unique identifiers are available. Therefore, a secure research environment and capable information technology support can directly influence the quality of the research data obtained and, by extension, research results. With faith in the integrity and security of the research environment, data providers may also be more likely to provide other unique data that can be important to driving truly innovative research.

A secure and well-performing environment is also important in that system performance and security controls can directly influence the scope of the research project, including the size and complexity of the data that can be managed and linked to support the project. Often, as the scope and complexity of research projects increase and the data volume grows, computing environments are challenged to scale up to ensure seamless operations.

This chapter describes key considerations concerning the research environment, including the technical platform and security considerations, to guide researchers as they seek to develop or optimize their systems for CER projects using large volumes of data such as linked registry and administrative claims data.

• Computing Systems and the Balance Between Security and Usability

As shown in Figure 2.1 , security and usability often stand on opposite ends of a spectrum. The tradeoff for having a highly secure system is decreased accessibility and practical usability, whereas systems that are highly accessible often face greater challenges in assuring data security. Understanding the scope of the research project and the needs of the researcher or research team is important to specifying a system configuration that meets the needs of the project and balances security with usability. For example, a single researcher with a small research project of limited scope will likely have different needs for a computing environment compared with a large decentralized research team undertaking a multiyear research study using national data.

Security versus usability.

We present three different computing system scenarios to help researchers identify where on the spectrum their program may fall. While this discussion does not take into account the number of users, it is important to note that the cost of large systems varies greatly depending on existing infrastructures and purchasing prices of solutions offered by various vendors.

Desktop Computer: High Security, Low Usability, Low Cost

In this environment, the user accesses all data on a dedicated desktop computer located in a dedicated and constantly locked office with limited network access.

• Security : The risks of theft and network attacks are reduced to a minimum.
• Usability : Multiple users will never be able to access the data concurrently.

Central Server: Medium Security, Medium Usability, Moderate Cost

This environment allows multiple users to connect remotely through a secure command line (e.g., through a secure protocol such as SSH) to a central computing server housing all the data and tools.

• Security : The risk increases while gaining access to information over a network. Controlling individual users’ access to information creates new administrative challenges.
• Usability : Multiple users can collaborate on a central system. Computing jobs can be submitted in the background and the progress can be checked from remote locations.

Virtual Remote Desktops: High Security, High Usability, High Cost

This environment allows multiple users to connect remotely to virtualized desktops over the Internet from laptops or desktop computers to access shared data and tools.

• Security : Since the accessing computers supply only monitor, keyboard, and mouse, the data never leave the server environment. Even secure printing to dedicated printers is possible to control paper output.
• Usability : Each user connects to a virtual computer in the central environment. All tools are housed on the server but accessed through existing desktops.
• Building the Technical Platform

Regardless of the selected technologies, the number of users, or security requirements, the technical platform can be disassembled into various components ( Figure 2.2 ). Defining the requirements for the individual components creates a meaningful information technology plan.

Component architecture.

Securing the Platform

Most regulatory security frameworks, such as the Health Information Portability and Accountability Act (HIPAA) and Federal Information Security Management Act (FISMA), focus on controlling the confidentiality, integrity, and availability of information. The efforts to implement administrative, physical, and technical safeguards tend to scale up as the system complexity increases. Regulatory requirements and risk assessments will strongly affect the technical implementations.

Users Accessing Platform

To support a wide range of innovative research on complex linked data resources, the experience and focus of team members narrow and deepen. Work is divided among individuals to cover areas such as data management, data linking, cohort discovery, advanced modeling, and more. Complex research projects depend on the seamless integration and collaboration of the various users and the use of their preferred tools. By defining the user profiles and job responsibilities, the main usability properties of the expected environment are established. Examples of typical roles within complex project teams include the following.

Role: Data Manager . The data manager takes care of the data. This might include importing new data, conversion of file formats, preparation and receipt of data carriers (storage devices), archiving obsolete data, and granting access to data.

Role: Data Linking . The linking expert is responsible for linking data sources. This might include cleaning linking variables, building linking methods, and cohort discovery (e.g., selection of patients meeting specific study-inclusion criteria), resulting in datasets for various research projects.

Role: Analyst/Statistician . The statistician is responsible for all modeling aspects of a research study. These might include creating analytic cohorts for study questions (using previously linked deidentified data), preparing data for modeling, and analyzing data to meet project objectives.

Gaining Access to Platform

Access management controls how users gain access to the system and data. An existing organization might have a central user-management process that establishes authentication with a simple username/password combination. More advanced two-factor authentication methods ensure that a compromised password alone does not convey access to the system. Biometric authentication (e.g., fingerprint reader) verifies the identity of the intended user. Examples of commonly used authentication methods and their pros and cons are provided in Appendix 2.1 .

Processing Power of Platform

The processing power of the computing system directly affects the time it takes to manipulate the data. As linking processes touch the same information repeatedly, tuning parameters to optimize the performance of hardware and software will reduce the run times. For considerations for hardware performance, see Appendix 2.1 .

It is important to understand that the researcher’s network has an impact on data flow. The network can quickly become the bottleneck for moving data, resulting in an exceptionally slow response. In an optimized setup, the connection between data and processing components is a dedicated Gigabit (1,000Mbps) network or even fiber optics. Any components between the processing and storage, such as firewalls, network switches, or routers, will reduce information flow. In a setup in which the data are stored on hard drives directly attached to the processing system, network performance will have a limited impact on data flow.

Data Storage Platform

The storage performance directly affects the time it takes to perform data tasks. For example, tasks such as cleaning and standardizing data, combining data sources, or performing exploratory analyses of linked data sources are storage-intensive activities. The main technical characteristics of the storage platform are size and speed. A storage device is attached using a specific technology, such as Serial Advance Technology Attachment (SATA), Serial Attached Small Computer System Interface (Serial SCSI or SAS), Universal Serial Bus (USB), or Storage Area Network (SAN). Various vendors sell enterprise storage solutions encapsulating multiple storage devices in a single appliance.

Storage Size . When purchasing data carriers, it is important to understand that physical data size and actual available data size will greatly vary depending on the installation. Methods used to prevent data loss, such as Redundant Array of Independent Disks (RAID), might require as much as twice the amount of physical space as required for storing the physical data. The file system used to store data also affects available data size. A data carrier is divided into blocks like a blank book with many pages. The size of the data block is fixed for the entire file system. As an example, if the block size is 1,024 characters (or bytes) and a file of 1,500 characters is saved, it will consume 2,048 physical bytes on the data carrier. Since the partially used blocks cannot be used for other files, these bytes are “lost.”

Storage Speed. Storage devices have two speed-related properties: (1) the time it takes to find the data on the carrier (referred to as seek time and measured in milliseconds) and (2) the continuous read/write performance. The seek time depends mainly on how fast the disk is spinning. Common rotation speeds are 5,400, 7,200, 10,000, and 15,000 revolutions per minute (RPM). In the case of a solid state disk (SSD), the seek time will be extremely low, as there are no moving parts. The continuous read/write performance depends not only on how fast the disk is spinning, but also on how the disk is attached to the processing system.

A well-performing storage system can read/write information at rates of 100MB/s or more. A disk spinning at 5,400 or 7,200 RPM, as delivered in standard laptops or desktops, generally cannot achieve this. In comparison, an SSD attached over SATA can easily reach read/write rates of 500MB/s or more. To optimize cost, a computing system can be outfitted with slower/cheaper storage for archiving in combination with fast analytic storage to support powerful processing.

• Securing the Research Environment

Federal and State laws have mandated several sets of regulations, each intended to address one or more of the following objectives:

• Confidentiality : Preserving authorized restrictions on information access and disclosure, including means for protecting personal privacy and proprietary information
• Integrity : Guarding against improper information modification or destruction; includes ensuring information nonrepudiation and authenticity
• Availability : Ensuring reliable and timely access to information

Regulatory Requirements

The research environment must comply with applicable laws to protect the hosted information. Because HIPAA governs health information collected by covered entities mainly during health encounters, alternative research datasets might require compliance with other contractual requirements or State regulations. Researchers should consult with a regulatory expert as early as possible to ensure that they understand the scope of all applicable laws.

Regulatory requirements generally describe what must be controlled and leave it up to the research team to define how to reach the required controls by implementing adequate policies and procedures. Some State-level privacy laws may govern even self-collected information. In the following sections, we describe a sample of regulatory requirements that might be applicable to researchers working with sensitive health information.

Federal Information Security Management Act of 2002 . FISMA defines a mandatory framework for managing information security for all information systems used or operated by a U.S. Federal Government agency or by a contractor or other organization on behalf of a Federal agency. It requires the development, documentation, and implementation of an information security program. The National Institute for Standards and Technology (NIST) standards and guidelines (Special Publications 800 series) and Federal Information Processing Standards (FIPS) publications further define the framework of the program.

Health Information Portability and Accountability Act . The U.S. Congress created HIPAA in 1996. Security standards establishing requirements to safeguard Protected Health Information (PHI), both paper and electronic (ePHI), were issued as part of HIPAA in April 2003. The security requirements specifically address administrative, physical, and technical safeguards meant to ensure that patient health records and Personally Identifiable Information (PII) remain as secure as possible.

State Security-Breach Laws . Forty-six States, the District of Columbia, and multiple U.S. territories (Guam, Puerto Rico, and the Virgin Islands) have enacted privacy-breach notification laws. While these laws can vary from State to State, they generally follow a similar framework. This framework defines “sensitive data,” sets out requirements for triggering the breach-notification process, identifies actors and roles in the notification process, defines to whom the law applies, and describes those cases under which certain parties and/or information may be exempt from notification requirements ( www.fas.org/sgp/crs/misc/R42475.pdf ). Researchers are responsible for understanding their responsibilities under the relevant State breach-notification legislation and should consult legislative resources such as the National Conference of State Legislatures for regulatory text ( www.ncsl.org ).

Identifying Sensitive Data

Sensitive data are the information protected by regulatory requirements. The definition of sensitive data varies widely between laws. In some cases, the scope of a Data Use Agreement (DUA) could even require aggregation or define minimum cell sizes. In the following section, we provide a summary of regulatory definitions per FISMA, HIPAA, and State security-breach laws.

Personally Identifiable Information, FISMA . As used in information security, PII is any information maintained by an agency that can be linked to an individual. This includes (1) any information (e.g., name, Social Security Number, date and place of birth, mother’s maiden name, or biometric records) that can be used to distinguish or trace an individual’s identity and (2) any other information (e.g., medical, educational, financial, and employment information) that is linked or linkable to an individual. Examples of PII include, but are not limited to—

• Name, such as full name, maiden name, mother’s maiden name, or alias
• Personal identification number, such as Social Security Number, passport number, driver’s license number, taxpayer identification number, or financial account or credit card number
• Address information, such as street address or email address
• Personal characteristics, including photographic image (especially of face or other identifying characteristic), fingerprints, handwriting, or other biometric data (e.g., retina scan, voice signature, facial geometry)

Protected Health Information, HIPAA . The HIPAA Privacy Rule protects all “individually identifiable health information” held or transmitted by a covered entity or its business associate in any form or media, whether electronic, paper, or oral. The Privacy Rule calls this information Protected Health Information , or PHI.

Under HIPAA, individually identifiable health information is information, including demographic data, that relates to any of the following:

• The individual’s past, present, or future physical or mental health condition
• The provision of health care to the individual
• The past, present, or future payment for the provision of health care to the individual
• Information that identifies the individual or for which there is a reasonable basis to believe it can be used to identify the individual

Individually identifiable health information includes many common identifiers (e.g., name, address, birth date, Social Security Number). The Privacy Rule excludes from PHI employment records that a covered entity maintains in its capacity as an employer, and educational and certain other records subject to or defined in the Family Educational Rights and Privacy Act, 20 U.S.C. §1232g.

Electronic Protected Health Information, HIPAA . The HIPAA Security Rule protects a subset of information covered by the Privacy Rule, which is all individually identifiable health information a covered entity creates, receives, maintains, or transmits in electronic form. The Security Rule calls this information electronic Protected Health Information , or ePHI. The Security Rule does not apply to PHI transmitted orally or in writing.

Limited Datasets, HIPAA . HIPAA also has a provision for Limited Datasets (LDSs) from which most but not all potentially identifying information has been removed. Elements in an LDS are often necessary for research; however, Direct Identifiers , a subset of PHI defined by HIPAA §164.514(e)(2), must be removed. The Direct Identifiers include—

• Postal address information other than town or city, State, and ZIP Code
• Telephone numbers
• Fax numbers
• Electronic mail addresses
• Social Security Numbers
• Medical record numbers
• Health plan beneficiary numbers
• Account numbers
• Certificate/license numbers
• Vehicle identifiers and serial numbers, including license plate numbers
• Device identifiers and serial numbers
• Web Universal Resource Locators (URLs)
• Internet Protocol (IP) addresses
• Biometric identifiers, including finger and voice prints
• Full-face photographic images and any comparable images

LDSs can include the following PHI:

• Date of birth
• Date of death
• Dates of service
• Town or city

Personal Information, State Security-Breach Laws . Researchers should review applicable State legislation for definitions of Personal Information. Generally, these definitions do not vary substantially from State to State and are very similar to Federal definitions. For example, the North Carolina State Security-Breach Laws (North Carolina General Statute §75-65) define Personal Information as a person’s first name or first initial and last name in combination with any of the following identifying information:

• Social Security Number or employer taxpayer identification numbers
• Driver’s license, State identification card, or passport numbers
• Checking account numbers
• Savings account numbers
• Credit card numbers
• Debit card numbers
• Personal Identification Number (PIN)
• Electronic identification numbers, electronic mail names or addresses, Internet account numbers, or Internet identification names
• Digital signatures
• Any other numbers or information that can be used to access a person’s financial resources
• Biometric data
• Fingerprints
• Parent’s legal surname before marriage

Summary of Protected Information

The research team may find it useful to summarize in matrix form the protected information types identified by applicable regulatory requirements. This matrix will help the research team identify information in datasets and assess the policies and procedures that might apply to a specific work task. Table 2.1 shows an example of one such matrix.

Example of matrix summarizing protected information types.

Building and Implementing a Security Plan

Meeting applicable regulatory requirements requires thoughtful planning and management. While it is tempting to think of information security in terms of technological controls, successful security management requires people, processes, and technology in equal proportion. An overarching security management plan addresses how people, processes, and technology will be leveraged to maintain the confidentiality, integrity, and availability of sensitive data within the bounds set by applicable regulatory requirements.

While development of the security management plan is an iterative process, with sections added or refined as planning activities proceed, the document will ultimately address the following:

• Security laws and regulations describe those regulatory requirements applicable to the research team, as discussed previously.
• Major functions list those functions the security plan is intended to accomplish.
• Scope lists those sensitive data types the security program is intended to address.
• Roles and responsibilities describe roles that will be held by members of the organization and their responsibilities vis-à-vis information security.
• Management commitment represents an official statement on the part of the applicable management body in support of the processes and procedures documented within the security plan.
• FISMA security categorization and impact level define the FISMA category assigned to the data and information systems covered by the security plan. This section is applicable only to those systems subject to FISMA.
• Compliance and entity coordination describe which roles are responsible for ensuring organizational compliance with the security plan and which roles are responsible for coordinating security activities among relevant entities external to the research team (e.g., data centers, overarching security offices).

Security documentation control

Risk management (described in further detail below)

Workforce security

Access management

Security training

Incident reporting

Contingency planning

Security assessment

Facility access

Workstation access

Devices and removable media

Data integrity

Authentication

Network security

System activity review/audit

At the outset of security planning, the research team should be able to define the security laws and regulations, major functions, and scope sections. Roles and responsibilities, management commitment, entity coordination, and FISMA categorization (if applicable) can be defined further through stakeholder meetings. The processes and procedures documented in the subplans will be developed as part of the risk-management process described below.

Workforce Training

A training plan defines working procedures, emergency and incident management, sanction policies, policies and procedures on how to inform members of the workforce about their roles and responsibilities, and other relevant procedures. Many large research environments might be able to leverage existing training modules. These might include training on HIPAA, research ethics, basic computer and network use, and basic human resources policies. Keeping the retraining on an annual basis is advisable.

• Managing Risks

Research teams must first understand regulatory requirements, then select and implement adequate security controls to meet these requirements and to mitigate risks posed to the security of the organization’s information systems and data. Often, discussions of information security mistakenly emphasize specific technical safeguards. An emphasis on risk management, however, properly defines technical solutions as the means by which organizational risks are controlled. Risk management, therefore, drives information security planning. A comprehensive risk-management program not only allows data custodians to identify risks posed to their data, but also provides a framework for selection of functional and technical security controls.

Data custodians subject to FISMA requirements should consult NIST guidance for implementing a FISMA-compliant life-cycle program, which includes detailed volumes of guidance and controls. We illustrate a more general risk-management framework in Figure 2.3 . This framework envisions risk management as a continuous cycle of assessing, addressing, and monitoring organizational risk to ensure the confidentiality, integrity, and availability of information systems.

Risk-management cycle.

Identifying and Assessing Risk

Risk identification is conducted on any technology, process, and procedure within the scope of the environment. Risk identification is, simply put, the process of identifying and documenting potential threats to the research team’s information and information systems. Risk identification can be conducted in a variety of ways, including brainstorming sessions, documentation reviews, assumptions analysis, cause and effect diagramming, strengths/weaknesses/opportunities/threats (SWOT) analysis, and expert consultation. Inclusion of an independent third party, be it an outside consultant or even representatives from a separate group within the research team, will provide an external point of view invaluable in fully defining the spectrum of potential adverse events. Regardless of the method used, this process must clearly identify and document the source of the risk and the impact of the risk should it be realized.

Assess identified risks along two primary dimensions: probability of occurrence and criticality of impact. Actions and mitigations planned in the next phase of the risk-management cycle will be based largely on each risk’s score as assessed during this phase.

Risk scores evaluate the combination of the probability and impact of a security breach/incident. Higher scores represent higher security risks. Lower scores represent reduced security risks. Table 2.2 provides an example of how to plot and assess the severity of a security breach/incident (criticality of impact) and the likelihood of occurrence of an event.

Example of risk scores.

Tracking Risks

The risk register is the collection of all identified risks, their assessed impact and probability, and possible actions/mitigations. Both HIPAA and FISMA mandate the analysis of risk and a record thereof. Table 2.3 is an example of a generic risk register.

Example of generic risk register.

Planning Risk Responses

Once risks have been identified, assessed, and documented in the risk register, data custodians and other stakeholders can plan appropriate methods of dealing with each risk. Risk responses can be divided into four categories: avoidance, acceptance, transfer, and mitigation.

• Risk avoidance occurs when a research team takes the necessary actions to reduce the likelihood of risk realization to close to (if not exactly) zero. Generally, risk avoidance is the most desirable method of dealing with risks; however, it is often cost prohibitive or simply infeasible to avoid all risks.
• Risk acceptance occurs when a research team chooses to accept the consequences of a risk should it be realized. Risk acceptance is generally recommended when the impact of a risk is small or when the probability of occurrence is significantly lower than the cost to avoid, transfer, or mitigate. Each research team must define its own criteria for what constitutes an “acceptable” risk.
• Risk transfer occurs when a research team passes the impact of risk realization on to another party. The most common form of risk transfer is insurance. Risk transfer is feasible only when the impact can be clearly measured and addressed by the third party.
• Risk mitigation occurs when a research team takes steps to reduce the probability or impact of risk realization. Risks that cannot be avoided, accepted, or transferred must be mitigated.

After a research team decides whether to avoid, accept, transfer, or mitigate a risk, it must determine the necessary steps to do so. At this juncture, the research team will identify the necessary and appropriate technical controls to either avoid or mitigate certain risks. Actions identified in this phase must also be documented in the risk register.

Implementing Risk Responses

During the implementation phase, the research team develops and deploys the technical controls identified in the planning phase. Just as importantly, the research team must also document the controls selected, develop all necessary records, and train stakeholders accordingly. Users must understand not only how to use any security controls implemented but also the “rules of behavior” for maintaining a secure environment. Technical controls alone are not sufficient to create a fully secure environment; users and other stakeholders must foster and maintain a culture of security.

Monitoring Risks

Risk management is an ongoing cyclical process. The research team must periodically reassess the environment for new or changing risks, which in turn must be identified, assessed, and addressed through planning and action. Thoughtful and frequent monitoring of risk allows a research team to adapt more easily to changes, both expected and unexpected, without compromising information security.

• Conclusions

A research environment incorporating a secure and well-performing computing platform represents the operational backbone for conducting innovative research using complex linked data sources. Securing and safeguarding the information not only meets legal and regulatory requirements, but also builds needed trust among stakeholders. Data providers will be more open to providing access to their information, researchers will be confident in accessing sensitive data, and programmers and analysts will operate effectively in a standardized environment with consistent application of technologies and tools. The technical implementation, combining performance and storage, will enable complex data management, as described in Chapter 3 . Supported by the leadership, successfully implemented security policies and procedures fit seamlessly into daily workflows, reducing and mitigating potential risks. The environment is now ready to support research and to receive even the most sensitive data.

Appendix 2.1. Procedures and Processes To Enhance Data Security

Moving sensitive data using cds or dvds.

Perform the following steps to create and transport sensitive data using CDs or DVDs. This procedure can be adapted for electronic transfer using a protocol such as the secure file transfer protocol (SFTP).

• Create a new media number and add it to a data carrier list tracking all movable media containing sensitive data.
• Create a local folder with the media number as the name.
• Assemble all the data in the created folder.
• Generate an encryption key using a GUID (globally unique identifier) tool such as that found at www.guidgenerator.com/ and print it on a document along with the media number.
• Create an archive using a PGP ( http://en.wikipedia.org/wiki/Pretty_Good_Privacy ) encryption tool with all the contents of the folder, using the GUID as the encryption key.
• After testing the self-extracting archive, use a file-shredding tool ( www.fileshredder.org/ ) to remove the folder with the data.
• Burn the archive onto a data CD or DVD labeled with the media number.
• You can now mail the data carrier, and fax or email the encryption key separately to the receiving party.

Guidelines for Storage and Destruction of Movable Media

• Store movable media in a safe, separated from the encryption keys.
• Destroy damaged and/or retired media, including hard disks, by shredding.
• Update movable media records for every media item that is disposed of or destroyed.
• Shred any printed material at location or use a secure document disposal service.

Decoupling and Mapping Data

Figure 2.4 describes a process for adding data sources to the research data environment, removing direct identifiers, and creating a merged dataset with elements regarding the individuals’ health status or health services utilization. In the context described below, we retain Protected Health Information (PHI) within the original data files but limit access (a process known as decoupling). When PHI is destroyed following the data linkage, this is known as deidentification. We use the terms deidentified and decoupled interchangeably in this report as we discuss data security and staging, but readers should understand the differences represented by the terminology.

Process for creating a merged dataset.

Data file : This file contains raw data such as claims, diagnoses, and treatment. Individuals cannot be directly identified in these data.

Finder file : The finder files contain direct identifiers of individuals.

Mapping process : Using a mapping method, the individuals in the finder files are matched.

Crosswalk : The crosswalk identifies matching records for each individual and possible duplicates. A new unique identifier (ID) is assigned for each true identified research subject.

Removal of direct identifiers : Using the crosswalk and the data file, a new file is generated by replacing the source identifiers with the new IDs.

Merged dataset : Information about an individual can now be accessed across data sources and deidentified datasets using the new IDs.

Physical Separation of Data Using Storage Architecture

In the computing environment, data are separated physically into the three storage areas, as shown in Figure 2.5 .

Physical storage areas.

Direct identifier storage : Raw data including direct identifiers are secured in a highly restricted part of the system. The direct identifier space is accessible only through very restrictive access-management controls. All work on data linking and removal of direct identifiers is performed in this environment. Only a limited set of specifically trained, authorized individuals work in this environment.

Deidentified storage : The deidentified space contains deidentified master datasets from each source. This storage might be accessible to authorized users in read-only mode. Research datasets are extracted, or “cut,” from these master files.

Project storage : Individual programmers access this space for projects. Institutional Review Boards and Data Use Agreements control the access to master datasets and datasets for projects.

Examples of Common Authentication Methods

Simple User Account . A user authenticates with username and password.

• Pros : This is a quick and simple way to access a system.
• Cons : Once the username and password are known, any individual can gain access.

Two-Factor Authentication . A user authenticates with username, password, and Personal Identification Number (PIN) (e.g., RSA secure ID).

• Pros : A changing PIN provides a second factor in addition to logging in with the username and password. An authentication is not possible without the device.
• Cons : The PIN verification is costly and, depending on the implementation, requires that the computing environment have access to the system verifying the PIN.

Biometric Authentication . A user authenticates with username, password, and (for example) a fingerprint.

• Pros : The additional biometric verification requires the account holder to be present at the time of authentication.
• Cons : Managing biometric information requires custom software installations on the client system.

Performance Considerations for Computer Hardware

Fine-tuning the three hardware components of memory, central processing unit (CPU), and bus speed is essential for the actual processing power. Random access memory (RAM) can be added after a purchase, but changes to the CPU or bus speeds are complicated and impractical. The bus speed represents how fast information can be moved between the CPU and the memory, and from and to the attached storage. When purchasing a system, you should purchase the highest affordable CPU/bus speed combination while leaving space to add memory later.

Central Processing Unit . Performance of a CPU is affected by (a) sockets (i.e., actual number of CPUs); (b) cores and threads (i.e., how instructions can be processed in parallel); and (c) clock (i.e., a number in GHz representing how many instructions are processed per second).

Random Access Memory . Selecting memory is dependent on the supported architecture of the motherboard. Manufacturers generally advise on what type of memory is supported and best for optimized performance. The product description of memory often includes physical and performance parameters. For example “240-Pin DDR3 1600” represents memory that has 240 pins connecting it to the motherboard and supports a 1600MHz clock.

Bus Speed . Depending on the architecture, there are multiple bus speeds affecting data throughput between storage, memory, and CPU. For example: a hard disk might be attached using SATA3 or “SATA 6Gb/s,” representing a bus speed of 6Gb/s. Bus speeds on the CPU/motherboard (chipset layout) are more complicated and fine-tuned by the vendor. Components with fancy names such as North Bridge, South Bridge, and Front Side Bus (FSB) ( http://en.wikipedia.org/wiki/Front-side_bus ) are part of this architecture. Vendors typically offer systems with high-performance options where these speeds are optimized and enhanced compared with home use products.

• Cite this Page Dusetzina SB, Tyree S, Meyer AM, et al. Linking Data for Health Services Research: A Framework and Instructional Guide [Internet]. Rockville (MD): Agency for Healthcare Research and Quality (US); 2014 Sep. 2, Research Environment.
• PDF version of this title (1.0M)

In this Page

• Procedures and Processes To Enhance Data Security

Other titles in these collections

• AHRQ Methods for Effective Health Care
• Health Services/Technology Assessment Texts (HSTAT)

Recent Activity

• Research Environment - Linking Data for Health Services Research Research Environment - Linking Data for Health Services Research

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Connect with NLM

National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894

Web Policies FOIA HHS Vulnerability Disclosure

Help Accessibility Careers

Environmental Meaning

Cite this chapter.

• Martin Krampen 3 , 4  

Part of the book series: Advances in Environment, Behavior, and Design ((AEBD,volume 3))

459 Accesses

4 Citations

In this chapter, advances in research on environmental meaning are addressed by connecting the present state of the art in research to the past, and by extrapolating from the present to possible future develop ments. Two concurrent approaches to the study of environmental meanings are addressed: semiotics and environmental psychology. The chapter concludes with a discussion of the ecological approach and its potential contribution to understanding environmental meaning.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime
• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Unable to display preview.  Download preview PDF.

Similar content being viewed by others

Hermeneutics, Place, and the Environment

The Problem of (with) Environmental Ethics

From Shared Enaction to Intrinsic Value. How Enactivism Contributes to Environmental Ethics

Agrest, D. (1977). Design versus non-design. Communications (Sémiotique de l’espace), 27 , 79–102.

Google Scholar  

Agrest, D., &Gandelsonas, M. (1973). Critical remarks on semiology and architecture. Semiotica ,9(3), 252–271.

Appleyard, D. (1969). Why buildings are known: A predictive tool for architects and planners. Environment and Behavior, 1 ,131–156.

Barthes, R. (1964). Eléments de sémiologie. Communications, 4 ,91–135.

Barthes, R. (1967). Elements of semiology (C. Smith and A. Lavers, Trans.). London: Cape.

Bauer, F. (1980). Zur Konzeptspezifität des Semantischen Differentials.-Eine Diskussionsbemerkung zu Flade’s: Die Beurteilung unweltpsychologischer Konzepte mit einem konzeptspezifischen und einem universellen Differential. Zeitschrift für experi mented una angewandte Psychologie, 27 ,163–167.

Bauer, F., &Bräunling, H. (1982). Ein Vergleich der Eignung konzeptspezifischer und universeller Formen des Semantischen Differentials zur Beurteilung von Urnweltausschnitten. Zeitschrift für experimented und angewandte Psychologie, 29(2) ,181–203.

Bechtel, R. B. (1980). Architectural space and semantic space: Should the twain try to meet. In G. Broadbent, R. Bunt, &T. Llorens (Eds.), Meaning and behavior in the built environment (pp. 215–222). New York: Wiley.

Beck, R. (1970). Spatial meaning and the properties of the environment. In H. M. Proshansky, W. H. Ittelson, &L. G. Rivlin (Eds.), Environmental psychology: Man and his physical setting (pp. 134–141). New York: Holt, Rinehart &Winston.

Berger, A., &Good, L. (1963). Architectural psychology in a psychiatric hospital. Journal of the American Institute of Architecture ,December, 76–80.

Berlyne, D. E. (1960). Conflict, arousal, and curiosity . New York: McGraw-Hill.

Blomeyer, G. R., &Helmholtz, R. M. (1976). Semiotics in architecture. Semiosis, 1 ,42–51.

Bortz, J. (1972). Beiträge zur Anwendung der Psychologie auf den Städtebau. II. Erkun-dungsexperiment zur Beziehung zwischen Fassadengestaltung und ihrer Wirkung auf den Betrachter. Zeitschrift für experimentelle und angewandte Psychologie, 19 ,226–281.

Boudon, P. (1973). Recherches sémiotiques sur le lieu. Semiotica ,7(3), 189–225.

Boudon, P. (1974). Définition (sémiotique) des critères pour une théorie des lieux: Architecture et méthodologies. MMI Bulletin des Séminaires Pédagogiques (Paris: Institut de l’Environnement), 4 ,3–57.

Boudon, P. (1977a). Introduction. Communications (Sémiotique de l’espace), 27 ,1–12.

Boudon, P. (1977b). Un modèle de la cité grecque. Communications (Sémiotique de l’espace), 27 ,122–167.

Boudon, P. (1978). Réécriture d’une ville: La Médina de Tunis. Semiotica ,22(1/2), 1–74.

Boudon, P. (1981a). Introduction à une sémiotique des lieux. Ecriture, graphisme, architecture . Montreal: Les presses de l’université de Montréal. Paris: Editions Klincksieck.

Boudon, P. (1981b). Recherches sémiotiques sur la notion de ’lieu architectural,’ Review Article. Recherches Sémiotiques/Semiotic Inquiry, 1 (4), 393–413.

Broadbent, G. (1969). Meaning into architecture. In C. Jencks &G. Baird (Eds.), Meaning in architecture (pp. 50–75). London: Barrie &Rockliff.

Broadbent, G. (1975). Function and symbolism in architecture. In B. Honikman (Ed.), Responding to social change . Stroudsburg, PA: Dowden, Hutchinson &Ross.

Broadbent, G. (1980a). The deep structures of architecture. In G. Broadbent, R. Bunt, &C. Jencks (Eds.), Signs, symbols, and architecture (pp. 119–168). New York: Wiley.

Broadbent, G. (1980b). Building design as an iconic sign system. In G. Broadbent, R. Bunt, &C. Jencks (Eds.), Signs, symbols, and architecture (pp. 311–331). New York: Wiley.

Broadbent, G. (1980c). A semiotic program for architectural psychology. In G. Broadbent, R. Bunt, &T. Llorens (Eds.), Meaning and behavior in the built environment (pp. 313–359). New York: Wiley.

Broadbent, G. (1980d). General introduction. In G. Broadbent, R. Bunt, &C. Jencks (Eds.), Signs, symbols, and architecture (pp. 1–4). New York: Wiley.

Broadbent, G., Bunt, R., &Llorens, T. (Eds.). (1980). Meaning and behavior in the built environment . New York: Wiley.

Broadbent, G., Bunt, R., &Jencks, C. (Eds.). (1980). Signs, symbols and architecture . New York: Wiley.

Castex, J., Depaule, J. C., &Panerai, P. (1978). Essai sur les structures syntaxiques de l’espace architectural. Notes méthodologiques en architecture et en urbanisme (Sémiotique de l’espace), 7, 101–155.

Castex, J., &Panerai, P. (1974). Structures de l’espace architectural. Notes méthodologiques en architecture et en urbanisme (Sémiotique de l’espace), 3/4 ,39–63.

Castex, J., &Panerai, P. (1979). Structures de l’espace architectural. In Sémiotique de l’espace (pp. 61–93). Editions Denoël/Gonthier.

Choay, F. (1970). Urbanism and semiology. In C. Jencks &G. Baird (Eds.), Meaning in architecture (pp. 26–37). London: Barrie &Rockliff.

Choay, F. (1970-1971). Remarques à propos de sémiologie urbaine. Larchitecture d’au-jourd’hui (La ville), 153 ,9–10.

Choay, F. (1973). Figures d’un discours méconnu. Critique (L’urbain et l’architecture), 311 , 293–317.

Duncan, J. S. (1985). The house as symbol of social structure: Notes on the language of objects among collectivistic groups. In I. Altman &C. M. Werner (Eds.), Home environments (pp. 133–151). New York: Plenum.

Eco, U. (1968). La struttura assente . Milano: Bompiani.

Eco, U. (1972). A componential analysis of the architectural sign/column. Semiotica ,5(2), 97–117.

Eco, V. (1980). A componential analysis of the architectural sign /column/. In G. Broadbent, R. Bunt, &C. Jencks (Eds.), Signs, symbols, and architecture (pp. 213–232). New York: Wüey.

Fauque, R. (1973). Pour une nouvelle approach sémiologique de la ville. Espace et sociétés, 9 , 15–27.

Fauque, R. (1979). Le discours de la ville. In S. Chatman, U. Eco, &J. M. Klinkenberg (Eds.), A semiotic landscape (pp. 918–923). Den Hague: Mouton.

Franke, J. (1969). Stadtbild: Zum Erleben der Wohnumgebung. Stadtbauwelt, 24 ,292–295.

Fusco, R. de (with M. L. Scalvini) (1960). Significanti e significanti nella rotonda palladiana. Op. cit. Selezione della critica d’arte conteni poranea, 16 ,5–23.

Fusco, R. de (1967). Architettura come mass-medium . Ban: Dedalo.

Gamberini, I. (1953). Per una analisi degli elementi dell’architettura . Firenze: Editrice Universitaria.

Gamberini, I. (1959). Gli elementi dell’architettura come ’parole’ del linguaggio architettonico . Firenze: Coppini.

Gamberini, I. (1961). Analisi degli elementi constitutivi dell’architettura . Firenze: Coppini.

Garroni, E. (1964). La crisi semantica della arti . Roma: Officina.

Garroni, E. (1972). Progetto di semiotica . Bari: Laterza.

Gibson, J. J. (1979). The ecological approach to visual perception . Boston: Houghton Mifflin.

Gibson, J. J. (1982). Notes on affordances. In E. Reed &R. Jones (Eds.), Reasons for realism: Selected essays of James J. Gibson (pp. 401–418). Hillsdale, NJ: Erlbaum.

Gottdiener, M. (1986). Urban culture. In T. A. Sebeok (Ed.), Encyclopedic dictionary of semiotics (Vol. 2, pp. 1141–1145). Amsterdam: Mouton de Gruyter.

Greimas, A. (1974). Pour une sémiotique topologique. Notes méthodologiques en architecture et en urbanisme (Sémiotique de l’espace). 3/4 ,1–21.

Greimas, A. (1979). Pour une sémiotique topologique. In Sémiotique de l’espace (pp. 11–43). Paris: Editions Denoel/Gonthier.

Groat, L. (1982). Meaning in post-modern architecture: An examination using the multiple sorting tool. Journal of Environmental Psychology, 2 ,3–22.

Groat, L., &Canter, D. (1979). Does post-modernism communicate? Progressive Architecture, 12 ,84–87.

Haeckel, E. (1866) Generelle Morphologie der Organismen ,Vol. 2: Allgemeine Entroucklungsgeschichte der Organismen . Berlin: Reimer.

Hall, E. T. (1966). The hidden dimension . New York: Doubleday.

Hammad, M. (1979). Sémiotique de l’espace et sémiotique de l’architecture. In S. Chatman, U. Eco, &J.-M. Klinkenberg (Eds.), A semiotic landscape (pp. 925–929). Paris: Mouton.

Harrison, D., &Howard, W. A. (1972). The role of meaning in the urban image. Environment and Behavior, 4 .

Hershberger, R. G. (1972). Toward a set of semantic scales to measure the meaning of architectural environments. In W. J. Mitchell (Ed.), Environmental design: Research and practice, Vol. 3 (pp. 6-4-1-6-4–10). Stroudsburg, PA: Dowden, Hutchinson &Ross.

Hillier, B., &Leaman, A. (1973). The man-environment paradigm and its paradoxes. Architectural Design, 8 ,507–511.

Hülier, B., &Leaman, A. (1975). The architecture of architecture. In D. Hawkes (Ed.), Models and systems in architecture and building (pp. 5–23). London: Construction Press.

Hillier, B., Leaman, A., Stansall, P., &Bedford, M. (1976). Space syntax. Environment and Planning B, 3 ,147–185.

Hjelmslev, L. (1968-1971). Prolégomènes à une théorie du langage . Paris: Les Editions de Minuit.

Honikman, B. (1973). Personal construct theory and environmental evaluation. In G. Broadbent, R. Bunt, &T. Llorens (Eds.), Meaning and behavior in the built environment (pp. 79–91). New York: Wüey.

Hunter, A. (1987). The symbolic ecology of suburbia. In I. Altman &A. Wandersman (Eds.), Neighborhood and community environments (pp. 191–221). New York: Plenum.

Jakobson, R. (1960). Closing statement: Linguistics and poetics. In T. A. Sebeok (Ed.), Style in language (pp. 350–377). Cambridge, MA: MIT Press.

Jencks, C. (Ed.). (1969). Meaning in architecture . London: Barrie &Rockliff.

Jencks, C. (1980). The architectural sign. In G. Broadbent, R. Bunt, &C. Jencks (Eds.), Signs, symbols, and architecture (pp. 71–118). New York: Wüey.

Kasmar, J. (1970). The development of a usable lexicon of environmental descriptors. Environment and Behavior, 2 ,153–169.

Kelly, G. A. (1955). The psychology of personal constructs . New York: Norton.

Koenig, G. K. (1964). Analisi del linguaggio architettonico . Firenze: Liberia Editrice Fiorentina.

Koenig, G. K. (1970). Architettura e communicazione . Preceduta da elementi di analisi del linguaggio architettonico. Firenze: Liberia Editrice Fiorentina.

Koffka, K. (1935). Principles of gestalt psychology . New York: Harcourt Brace.

Krampen, M. (1971). Das Messen von Bedeutung in Architektur, Stadtplanung und Design. Teil 1: Das Polaritätsprofil als Meßinstrument. Werk, 1 ,57–60.

Krampen, M. (1974). A possible analogy between (psycho-) linguistic and architectural measurement-the type-token ration (TTR). In D. Canter &T. Lee (Eds.), Psychology and the built environment (pp. 87–95). London: Architectural Press.

Krampen, M. (1979a). Meaning in the urban environment . London: Pion.

Krampen, M. (1979b). Survey on current work in semiology of architecture. In S. Chatman, U. Eco, &J.-M. Klinkenberg (Eds.), A semiotic landscape (pp. 169–194). The Hague: Mouton.

Krampen, M. (1980). The correlation of “objective” facade measurements with subjective facade ratings. In G. Broadbent, R. Bunt, &T. Llovens (Eds), Meaning and behavior in the built environment ,(pp. 61–78). New York: Wiley.

Krampen, M. (1989). Semiotics in architecture and industrial product design. Design Issues , 5(2), 124–140.

Lagopoulos, A.-P. (1978). Analyse sémiotique de l’agglomération européenne pré-capitaliste. Semiotica ,23(1/2), 99–164

Lagopoulos, A.-P. (1986). Settlement space. In T. A. Sebeok (Ed.), Encyclopedic dictionary of semiotics (Vol. 2, pp. 924–936). Amsterdam: Mouton de Gruyter.

Lawrence, R. J. (1985). A more humane history of homes: Research method and application. In I. Altman &C. M. Werner (Eds.), Home environments (pp. 113–132). New York: Plenum.

Ledrut, R. (1970). L’image de la ville. Espaces et sociétés, 1 ,93–106.

Ledrut, R. (1973). Parole et silence de la ville. Espaces et sociétés, 9 ,3–14.

Lewin, K. (1926). Untersuhungen zur Handlungs-und Afekt-Psychologie, I, II. Psychol-ogische Forschung, 7 ,294–385.

Lingoes, J. C. (1973). The Guttman-Lingoes nonmetric program series . Ann Arbor, MI: Mathesis Press.

Lynch, K. (1960). The image of the city . Cambridge, MA: MIT Press.

Miller, G. A. (1967). Psycholinguistic approaches to the study of communication. In D. L. Arm (Ed.), Journeys in science (pp. 22–73). Albuquerque: University of New Mexico Press.

Miller, G. (1969). A psychological method to investigate verbal meaning. Journal of Mathematical Psychology ,6, 169–191.

Moore, G. T. (1979). Knowing about environmental knowing. Environment and Behavior, 11 , 33–70.

Moore, G. T., &Golledge, R. G. (Eds.). (1976). Environmental knowing: Theory, research, and methods . New York: Van Nostrand Reinhold.

Morris, C. W. (1964). Signification and significance . Cambridge, MA: MIT Press.

Mukarovsky, J. (1978). On the problem of function in architecture. In J. Burbank &P. Steiner (Eds.), Structure, sign, and function: Selected essays by Jan Mukarovsky (pp. 236– 250). New Haven: Yale University Press.

Nöth, W. (1985). (Ed.). Architektur. In Handbuch der Semiotik (pp. 400–408). Stuttgart: J. B. Metzlersche Verlagbuchhandlung.

Osgood, C. E., Suci, G. J., &Tannenbaum, P. H. (1957). The measurement of meaning . Urbana: University of Illinois Press.

Ostrowetsky, S., &Bordreuil, S. (1979). Sociologie et sémiotique. In S. Chapman, U. Eco, &J.-M. Klinkenberg (Eds.), A semiotic landscape (pp. 956–959). Den Hague: Mouton.

Piaget, J., &Tuhelder, B. (1948). La représentation de l’espace chez l’enfant . Paris: Presses universitaire de France. (English translation by F. J. Langdon &J. L. Lunzre: The child’s conception of space . London: Routledge &Kegan Paul, 1956.)

Pinxten, R. (1974). Emicism and how to avoid a paradox. Communication and Cognition , 7(3/4), 315–333.

Pinxten, R. (1976). Epistemic universale: A contribution to cognitive anthropology. In R. Pinxten (Ed.), Universalism and relativism in language and thought (pp. 117–175). Den Hague: Mouton.

Pinxten, R. (1977). Descriptive semantics and cognitive anthropology: In search for a new model. Communication and Cognition ,10(3/4), 89–106.

Posner, R. (1986). Zur Systematik der beschréibung verbaler und nonverbaler kom-munikatíon. In, H. G. Bosshardt (Ed.), Perspektiven auf sprache (pp. 267–313)

Walter de Gruyter. Preziosi, D. (1979a). Architecture, language, and meaning . Den Hague: Mouton.

Preziosi, D. (1979b). The semiotics of the built environment: An introduction to architectonic analysis . Bloomington: Indiana University Press.

Preziosi, D. (1983a). The network of architectonic signs. In T. Borbé (Ed.), Semiotics unfolding (Vol. 3, pp. 1343–1349). Den Hague: Mouton.

Preziosi, D. (1983b). Minoan architectural design. Berlin: Mouton.

Preziosi, D. (1986). Architecture. In T. A. Sebeok (Ed.), Encyclopedic dictionary of semiotics (Vol. 1, pp. 44–50). Amsterdam: Mouton de Gruyter.

Proshansky, H. M., Ittelson, W. H., &Rivlin, L. G. (1970). Environmental psychology: Man and his physical setting . New York: Holt, Rinehart &Winston.

Rapoport, A. (1982). The meaning of the built environment: A nonverbal communication approach . Beverly Hills, CA: Sage.

Renier, A. (1979). Nature et lecture de l’espace architectural. In Sémiotique de l’espace (pp. 44–59). Paris: Editions Denoël/Gonthier.

Rossi-Landi, F. (1968). Il linguaggio come lavoro e come mercato . Milano: Bompiani.

Rossi-Landi, (1972). Omologia della riproduzione sociale. Ideologia, 16/17 ,43–103.

Rossi-Landi, F. (1975). Linguistics and economics . Den Hague: Mouton.

Ruesch, J. &Kees, W. (1970). Function and meaning in the physical environment. In H. M. Proshansky, W. H. Ittelson, &L. G. Rivlin (Eds.), Environmental psychology: Man and his physical setting (pp. 141–153). New York: Holt, Rinehart &Winston.

Sanchez-Robles, C. (1980). The social conceptualization of home. In G. Broadbent, R. Bunt, &T. Llorens (Eds.), Meaning and behavior in the built environment (pp. 113–133). New York: Wiley.

Scalvini, M. L. (1968). Per una teoria dell’architettura. Op. Cit.: Selezione della critica d’arte contemporanea, 13 ,30–44.

Scalvini, M. L. (1975). L’architettura come semiotica connotative . Milano: Bompiani.

Scalvini, M. L. (1979). A semiotic approach to architectural criticism. In S. Chatman, U. Eco, &J.-M. Klinkenberg (Eds.), A semiotic landscape (pp. 965–969). New York: Mouton.

Scalvini, M. L. (1980). Structural linguistics versus the semiotics of literature: Alternative models for architectural criticism. In G. Broadbent, R. Bunt, &C. Jencks (Eds.), Signs, symbols, and architecture (pp. 411–420). New York: Wiley.

Sebeok, T. A. (Ed.). (1986). Encyclopedic dictionary of semiotics (Vols. 1-3). Den Hague: Mouton.

Sommer, R. (1965). The significance of space. Journal of the American Institute of Architecture , May, 63-65.

Stringer, P. (1980). The meaning of alternative future environments for individuals. In G. Broadbent, R. Bunt, &T. Llorens (Eds.), Meaning and behavior in the built environment (pp. 93–111). New York: Wiley.

Stokols, D., &Altman, I. (Eds.). (1987). Handbook of environmental psychology . New York: Wiley.

Uexküll J. von (Ed.). (1913). Tierwelt oder Tierseele. In Bausteine zu einer biologischen Weltanschauung (pp. 77–100). München: Bruckmann.

Walther, E. (1974). Allgemeine Zeichenlehre. Einfuhrung in die Grundlagen der Semiotik . Stuttgart: Deutsch Verlagsanstalt.

Wapner, S., Kaplan, B., &Cohen, S. B. (1980). An organismic-developmental perspective for understanding transactions of men and environments. In G. Broadbent, R. Bunt, &T. Lloren (Eds.), Meaning and behavior in the built environment (pp. 223–255). New York: Wiley.

Warren, W. J. (1984). Perceiving affordances: Visual guidance of stair climbing. Journal of Experimental Psychology: Human Perception and Performance, 10 ,683–703.

PubMed   Google Scholar  

Wohlwill, J. F. (1976). Environmental aesthetics: The environment as a source of affect. In I. Altman &J. F. Wohlwill (Eds.), Human behavior and environment (Vol. 1, pp. 37–86). New York: Plenum.

Wohlwill, J. F. (1977). Environmental psychology: An overview. In B. B. Wolman (Ed.), International encyclopedia of psychiatry, psychology, psychoanalysis &neurology (Vol. 4, pp. 338–341). New York: Aesculapius.

Wolman, B. B. (Ed.). (1977). International encyclopedia of psychiatry, psychology, psychoanalysis &neurology (Vol. 4). New York: Aesculapius.

Zube, E. H. (1976). Perception of landscape and land use. In I. Altman &J. F. Wohlwill (Eds.), Human behavior and environment (Vol. 1, pp. 87–121). New York: Plenum.

Download references

Author information

Authors and affiliations.

University of the Arts Berlin, Federal Republic of Germany

Martin Krampen

Am Hochstrasse 18, D-7900, Ulm-Donau, Federal Republic of Germany

You can also search for this author in PubMed   Google Scholar

Editor information

Editors and affiliations.

School of Renewable Natural Resources, University of Arizona, Tucson, Arizona, USA

Ervin H. Zube

School of Architecture and Urban Planning, University of Wisconsin-Milwaukee, Milwaukee, Wisconsin, USA

Gary T. Moore

Rights and permissions

Reprints and permissions

Copyright information

© 1991 Plenum Press, New York

About this chapter

Krampen, M. (1991). Environmental Meaning. In: Zube, E.H., Moore, G.T. (eds) Advances in Environment, Behavior, and Design. Advances in Environment, Behavior, and Design, vol 3. Springer, Boston, MA. https://doi.org/10.1007/978-1-4684-5814-5_7

Download citation

DOI : https://doi.org/10.1007/978-1-4684-5814-5_7

Publisher Name : Springer, Boston, MA

Print ISBN : 978-1-4684-5816-9

Online ISBN : 978-1-4684-5814-5

eBook Packages : Springer Book Archive

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

• Undergraduate study
• Find a course
• Open days and visits
• New undergraduates
• Postgraduate study
• Find a programme
• Visits and open days
• New postgraduates
• International students
• Accommodation
• Schools & faculties
• Business & partnerships
• Current students
• Current staff

Academic Quality and Policy Office

• Academic Integrity
• Academic Student Support
• Assessment and Feedback
• Committees and groups
• Degree Outcomes Statement
• Educational Partnerships
• Guidance for Doctoral Training Entities
• Guidance for PGR supervisors
• PGR examiners' guidance
• Area A: Contents page
• Area B: PGR programmes, registration and the period of study
• Area C: PGR student entitlements and responsibilities 
• Area D: PGR skills development and the research environment
• Area E: PGR supervision
• Area F: PGR progress and review arrangements
• Area G: PGR dissertations, examinations, and outcomes 
• Programme and Unit Development and Approval
• Quality Framework
• Student Surveys
• Undergraduate Education
• Unit feedback from students

Related links

• Education and Student Success
• Bristol Institute For Learning and Teaching
• QAA Quality Code

Education and Student Success intranet

University home > Academic Quality and Policy Office > Postgraduate Education > Regulations and code of practice for research degree programmes > Area D: PGR skills development and the research environment

PGR skills development and the research environment

The regulations in this section set out the requirements for supporting PGR students in developing their skills and having access to an appropriate research environment.

On this page

Support for pgr student development, minimum requirements for skills development, expectations on access to the research environment.

The policy on PGR personal and professional development also relates to this section.

8.1. The University recognises the importance of training and development opportunities for PGR students within a high-quality research environment. These opportunities can enhance a PGR student’s effectiveness as a researcher and can underpin their subsequent career.

8.2. A PGR student’s training and development opportunities must be tailored to their needs, and will include activities provided by schools, faculties, and the personal and professional development programme . Some training and development opportunities might be provided by external sources.

8.3. Supervisors must provide guidance and support for PGR students on training and development opportunities with the expectation that the student will progressively take ownership of their own personal and professional development.

8.4. A PGR student must have access to relevant training and development opportunities in research skills and techniques, as well as in wider personal and professional development.

8.5. Supervisors must consider their PGR student’s training and development needs and assist them in identifying relevant activities at the beginning of the student’s period of study. Supervisors and the student must regularly review the student’s training and development needs.

8.6. Funded PGR students must complete any specific training required by their funder. The supervisors and student must ensure that any funder requirements for training are met within an appropriate timeframe.

8.7. The University provides a high-quality research environment in which PGR students develop their skills and conduct work on their research projects.

8.8. Schools and faculties must ensure that PGR students have access to an appropriate research environment, including the following:

8.8.1. Opportunities to interact with research-active staff in the student’s research area within the University and more widely.

8.8.2. Opportunities to experience and contribute to research activities within the school and faculty, such as presenting research at school seminars.

8.8.3. Access to any necessary facilities or resources to support the student’s work. PGR students who are working remotely must retain access to any required facilities or resources.

8.8.4. Access to any external facilities, resources, or expertise that is required for the student’s work and that cannot be provided from within the University.

University of Bristol Beacon House Queens Road Bristol, BS8 1QU, UK Tel: +44 (0)117 928 9000 Contact us

Information for

• New students

Connect with us

Study at bristol.

• Students' Union
• Sport, exercise and health
• Find a researcher
• Faculty research
• Impact of our research
• Research quality and assessment
• Engaging with the public

About the University

• Maps and travel
• Tours and visits
• The University on film
• Explore the city of Bristol
• Board of Trustees

Support the University

• Alumni and friends
• Working at Bristol
• Job listings

A–Z of the University

• Terms and conditions
• Accessibility statements
• Privacy and cookie policy
• Modern Slavery statement
• © 2024 University of Bristol

Today's Climate

Fossil fuel funding is ‘embedded’ across academia. what does that mean for climate research, oil and gas companies often help fund climate research on campuses. but these ties could pose major—and often underreported—conflicts of interest, new research finds..

Share this article

28 Million Acres of Alaska Public Lands Protected from Oil Drilling Following Trump-Era Reversal

When a Glacier Melts, What Does It Leave Behind?

After a Quiet Start, Climate Gets More Attention as the DNC Wraps Up

In the most extensive analysis of its kind, new research suggests that fossil fuel influence is widespread across universities in the United States, United Kingdom, Canada and Australia. 

Oil and gas companies have poured funding into campuses for decades. But scientists, journalists and students are only just starting to uncover the true extent of these financial ties—and how potential conflicts of interest in higher education could hinder efforts to combat climate change, the study’s authors say. 

“It’s a really troubling lack of transparency that kind of has created this situation where people have been trying to pull back the curtain on some of this, but struggling because a lot of this data just is not in the public domain,” study co-author Geoffrey Supran , an associate professor of environmental science and policy at the University of Miami, told me. “We observe that fossil fuel companies have embedded themselves widely within universities.” 

In the past few years, student activists have increasingly pushed their universities to divest from oil and gas on campus and in investment portfolios. Now, this movement is trickling into the university research community amid a growing push to increase transparency of fossil fuel funding sources—and potentially cut ties altogether. 

Fossil Fuel Funding: Supran has firsthand experience with fossil fuel money permeating the research space. The first year of his doctoral studies at the Massachusetts Institute of Technology was funded by an oil company. 

“They took us to fancy Italian banquets, they gave us free stationery with their logos on, they funded the first year of my Ph.D. And so my only association with them was positive,” Supran said. He explained that this type of treatment could result in reciprocity bias, which is when someone may feel the expectation to return favors after receiving gifts or incentives.

“It wasn’t until I started to pay more attention to the oil industry’s political machinations that I started to open my eyes,” he said. 

MIT did not respond to a request for comment about how the university mitigates this type of bias. 

Supran noted that “conflicts of interest are not necessarily implied bias.” However, a 2022 study published in the journal Nature Climate Change found university research centers funded by fossil fuel companies were more supportive of natural gas than those that are not. 

The new study finds a dearth of research investigating other potential ways that fossil fuel funding can influence climate research. As part of their work, the scientists parsed through around 14,000 peer-reviewed articles about conflicts of interest, bias and research funding across all industries. Just seven discussed fossil fuels. 

But their own analysis of literature, news reports and other sources revealed hundreds of instances of fossil fuel ties on campuses—from industry representatives sitting on governing research boards to fossil fuel-sponsored scholarships, internships and field trips for students. Some experts argue that these types of university partnerships could help fossil fuel companies “greenwash” their image. 

In other cases, oil and gas companies could have outsize control over what types of climate research occurs, such as ExxonMobil’s influence on carbon capture projects at Louisiana State University, which The Guardian and The Lens reported on in April . 

In March, my colleague Phil McKenna wrote about a new climate change initiative at MIT’s Sloan School of Management, and some of his sources noted their concern that the school could seek future funding from fossil fuel companies, as it has before with other projects. MIT’s Energy Initiative, a separate research center dedicated to developing low-carbon solutions, has raised more than $1 billion for energy research since 2006, approximately 45 percent from oil and gas companies, a spokesperson for the MIT Energy Initiative told Inside Climate News. 

Other fields have faced similar scrutiny from the public for industry ties, particularly in the biomedical and tobacco sectors. 

Shining Light on Financial Ties: With greenhouse gas emissions continuing to rise, students across many campuses are demanding that their universities drop all direct investments in fossil fuels. Since this movement began, more than 200 educational institutions have pledged to divest, including New York University and Dartmouth College. 

Now, there is a call to action from experts in the climate space to impose policies that ban fossil fuel influence on university research, which University of California, San Diego’s Craig Callender calls “divestment 2.0.” 

“Before, it was divesting the portfolio of the university. Now it’s looking at all these entanglements throughout the university and wanting to disassociate in this way as well,” Callender, who studies ethics and philosophy in science and was not involved in the new research, told me. “This [new study] proves beyond a shadow of a doubt that this knowledge institution is being weaponized against the public good.”

In 2022, Callender wrote an op-ed for The Chronicle of Higher Education about how fossil-fuel funding is influencing university research articles in favor of oil and gas. Academic studies are often cited in efforts to enact energy policies in government. More than 750 academics signed a letter in 2022 pushing for a ban on fossil-fuel funding for climate research. 

However, pulling this funding could have widespread consequences for the universities that rely on it. Over the past decade, state governments have invested significantly less money on research at public colleges and universities than in the past, forcing many institutions to find the money elsewhere. Instead of a full-scale ban, some schools, such as UC San Diego, are pursuing policies that require public disclosure of all external funding, including from oil and gas. 

However, Supran said these efforts aren’t happening fast enough.

“We also have observed a kind of worrying delay that has occurred between when civil society initially began to raise the alarm about this problem in the early 2000s until when scholars, and especially university leaders, have begun to pay attention to this issue,” he said. 

More Top Climate News 

The stats are in: This summer was the hottest on record in the Northern hemisphere, according to the European Union’s Copernicus Climate Change Service. The average global temperature over the past three months was 1.24 degrees Fahrenheit hotter than the 1991-2020 average. There was also a revolving door of scorching extreme weather events and catastrophes—from devastating heat waves in Europe to fires that burned through California. 

In May, the National Oceanic and Atmospheric Administration predicted above-normal hurricane activity this season. But the past three weeks have been unusually quiet on the hurricane front in the Atlantic —in what is typically the throes of this season. This has scientists wondering if the forecast was wrong or if we are in for a late season blitz over the next month, Judson Jones reports for The New York Times .

Attacked by Yemen’s Houthi rebels, a burning oil tanker is idling in the Red Sea—and emergency workers ditched an initial effort to tow it away due to poor conditions , Jon Gambrell reports for The Associated Press . This could represent a looming ecological disaster: Experts say more damage on the boat could trigger one of the worst oil spills in recent history. 

“The onus is on the Houthis, again, to look at the impact that they’re having, not only in the short term, but on the long term as it relates to the environment, the economy and the safety of those that are transiting this important waterway,” U.S. Air Force Maj. Gen. Pat Ryder, the Pentagon’s press secretary, said in a statement . 

On a visit to Istiqlal Mosque in Jakarta, Pope Francis issued a joint statement with Grand Imam Nasaruddin Umar calling on Muslims and Catholics to push for “decisive action” in the face of climate change . 

“The human exploitation of creation, our common home, has contributed to climate change, leading to various destructive consequences such as natural disasters, global warming and unpredictable weather patterns,” the statement reads. The people of Jakarta are intimately familiar with the impacts of climate change as the city is quite literally sinking into the ocean while simultaneously being swallowed by rising sea levels.

As winters get hotter with climate change, ski resorts are hoarding stockpiles of snow to use during the peak season , Chris Baraniuk writes for Wired . To do this, owners are stashing the icy mounds under insulating blanket systems developed by companies that say the products can prevent melting even during hot summer days.  

About This Story

Perhaps you noticed: This story, like all the news we publish, is free to read. That’s because Inside Climate News is a 501c3 nonprofit organization. We do not charge a subscription fee, lock our news behind a paywall, or clutter our website with ads. We make our news on climate and the environment freely available to you and anyone who wants it.

That’s not all. We also share our news for free with scores of other media organizations around the country. Many of them can’t afford to do environmental journalism of their own. We’ve built bureaus from coast to coast to report local stories, collaborate with local newsrooms and co-publish articles so that this vital work is shared as widely as possible.

Two of us launched ICN in 2007. Six years later we earned a Pulitzer Prize for National Reporting, and now we run the oldest and largest dedicated climate newsroom in the nation. We tell the story in all its complexity. We hold polluters accountable. We expose environmental injustice. We debunk misinformation. We scrutinize solutions and inspire action.

Donations from readers like you fund every aspect of what we do. If you don’t already, will you support our ongoing work, our reporting on the biggest crisis facing our planet, and help us reach even more readers in more places?

Please take a moment to make a tax-deductible donation. Every one of them makes a difference.

David Sassoon Founder and Publisher

Vernon Loeb Executive Editor

Kiley Price

Kiley Price is a reporter at Inside Climate News, with a particular interest in wildlife, ocean health, food systems and climate change. She writes ICN’s “Today’s Climate” newsletter, which covers the most pressing environmental news each week.

She earned her master’s degree in science journalism at New York University, and her bachelor’s degree in biology at Wake Forest University. Her work has appeared in National Geographic, Time, Scientific American and more. She is a former Pulitzer Reporting Fellow, during which she spent a month in Thailand covering the intersection between Buddhism and the country’s environmental movement.

• @kileyjprice
• [email protected]

Newsletters

We deliver climate news to your inbox like nobody else. Every day or once a week, our original stories and digest of the web's top headlines deliver the full story, for free.

• Inside Clean Energy
• Breaking News
• I agree to the terms of service and privacy policy .

By Kiley Price

Most Popular

Hope for North America’s Most Endangered Bird

By amy green.

Customers Sue an Arizona Water District Amid Drought and Surging Demand.

By wyatt myskow.

The Deteriorating Environment Is a Public Concern, but Americans Misunderstand Their Contribution to the Problem

By katie surma.

The U.S. Department of the Interior reaffirmed protection for wide swaths of Alaska’s land and water, a win for many Alaska Native peoples.

Keep Environmental Journalism Alive

ICN provides award-winning climate coverage free of charge and advertising. We rely on donations from readers like you to keep going.