Tutorial: Evaluating Information: Gray Literature

What is Gray Literature?

Gray (or grey) literature is literature produced by individuals or organizations outside of commercial and/or academic publishers. This type of non-formally published substantive information (often not formally peer-reviewed; especially important in all kinds of sciences) can include information such:

• theses and dissertations
• technical reports 
• working papers 
• government reports
• evaluation and think tank reports and resources
• conference proceedings, papers and posters
• publications from NGOs, INGOs, think tanks and policy institutes
• patents
• preprints
• unpublished clinical trials
• and much more

The sources you select will be informed by your research question and field of study, but should likely include, at a minimum, theses and dissertations.

Why Search the Gray Literature?

Most of gray literature is considered less prestigious, reliable, and "official" than publication in a peer-reviewed journal. But they are still fully legitimate avenues of publication. Often they are used to publicize early findings, before a study is entirely complete. Or, in the case of theses, they are published as a condition of receiving an advanced degree. Government technical reports are issued either by agencies that do scientific research themselves or else by a lab that has received government funding. Increasingly, such labs may be required to publish technical reports as a condition of receiving such funding. Gray literature may be cited like any other paper although with the caveat mentioned before that it is considered less "official" and reliable than peer-reviewed scientific papers.

When doing evidence synthesiis, it's important because the intent is to synthesize all available evidence that is applicable to your research question. There is a strong bias in scientific publishing toward publishing studies that show some sort of significant effect. Meanwhile, many studies and trials that show no effect end up going unpublished. But knowing that an intervention had no effect is just as important as knowing that it did have an effect when it comes to making decisions for practice and policy-making. While not peer-reviewed, gray literature represents a valuable body of information that is critical to consider when synthesizing and evaluating all available evidence.

• Identify and record the sources you will search. The sources you search will be informed by your research question and where you expect to find information related to your question.
• Document where you are searching and your search strategies, including document resource name, URL, search terms, and date searched.
• Collect citation information as you go.
• Adhere to your established inclusion and exclusion criteria when selecting sources.

What are technical reports? Technical reports present facts and conclusions about technological designs and projects. Typically, a technical report includes research about concepts as well as graphical depictions of designs and data. A technical report also follows a strict organization. Thus, when engineers read it, they can quickly locate the information they need.1

A technical report is a form of grey literature: Reports or documents produced by academia, government, industry or nonprofit organizations describing their work, proposals and/or the challenges they face. Grey literature is copyrighted like most other published work, but it usually is not distributed by commercial publishers the way a book or periodical would be but rather by the authoring organization itself. The organization that publishes the report is usually considered the 'author' (even if named individuals are credited within the report).

Some sources of technical reports:

Working papers are preliminary works, released to share ideas or invite discussion and feedback, often prior to the submission of a paper to a peer-reviewed journal or conference. Economists have traditionally relied on working papers as a method for the timely and informal communication of recent research findings.

Finding gray literature and searching it systematically is challenging. But there are a few approaches that you can take to add some structure to your search of this type of information:

• Search databases that specialize in gray literature: See the box below for more information.
• Search for theses and dissertations: There are a number of databases dedicated to theses and dissertations, which you can search using your search terms. See the box below for links to these resources. 
• Search clinical trials: There may be clinical trials being conducted that are relevant to your research question, but that haven't been published yet or never were published. See the box below for links to these resources. 
• Identify government agencies and international and non-governmental organizations that might publish technical papers and reports on your topic. Search their websites or any online libraries that they may provide. See the box below for links to some examples.
• Search conference proceedings and newsletters: Identify professional organizations that have and/or conferences at which researchers might be presenting work related to your topic. Search those conference proceedings or newsletters on the organization's website or by contacting organizational boards for access to past proceedings that may not be online. See the box below for some examples.
• Contact known researchers in the field to determine if there are any ongoing or unpublished studies that s/he may be aware of.
• Search professional and trade magazines. Professional magazines contain literature that is written by professionals in the field for other professionals in the field, but that may not be about research. Trade magazines contain advertisements and news very specific to a topic or industry.

Not all websites are created equal! Major nongovernmental organizations and various scholarly associations can be rich and authoritative websites, especially for the kind of grey literature not published in mainstream scholarly journals (including white papers, evaluation reports, training materials, and the like). Below are a few useful places to start.