EAHE 547: Evaluating Educational Research: Home

Evaluating Articles is Complicated

This guide will provide you with some of the tools it helps to be aware of in order to make your life easier. Below is information on tools like the H-Index, citation counts, and journal citation reports. First, however, here are some tips from THE Journal of things to look for in the articles you choose:

Educational relevance. The research should address interventions, outcomes, participants and settings representative of the school's interests and needs.

Rigorous, systematic and objective methods. The research should offer the highest quality evidence of what really caused the changes in the outcomes measured. According to the Education Department, the best way to produce such evidence is to conduct an experiment, referred to by some as "the gold standard" of research.

Sufficient detail for replication. The research methods and instruments should be described in enough detail that other researchers can replicate the study.

Submitted to independent, expert review. There should be evidence that the research was reviewed by research and content experts other than the researchers. A typical form of expert review is publication in a refereed journal.

For example…

Take the retracted paper "Ileal-lymphoid-nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in children" as an example. Although submitted to two rounds of peer-review, this article on the MMR vaccine contained falsified information and was never able to be replicated. This not-so-scientific paper is a perfect example of how sometimes faulty research can get through the peer review process and still be published in one of the highest ranked academic journals. Andrew Wakefield, the "father" of the anti-vaccine movement produced this unethical paper that had serious repercussions.

What is it?

The h-index measures the research impact of an individual scholar. Web of Science, Scopus, and Google Scholar can also be used to calculate an h-index. An h-index of 20 means there are 20 items that have 20 citations or more. This metric is useful because it discounts the disproportionate weight of highly cited papers or papers that have not yet been cited.

Finding the H-Index in Web of Science

1. Go to Web of Science. You can use this link if you have it OR (if you do not have this link) from the library homepage, click "Articles & Databases," then "Web of Science" under "Jump to your favorite database." From the menu that appears, select Cited Reference Search.
    
2. Enter the author's name, the work's source, and/or the publication year for the work that you would like to see who has cited it. (You can use Ileal-lymphoid-nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in children if you'd like to try a sample search. Copy and Paste this and search for it as Title.)


3. Click on "Search" to retrieve the scholarly works that match your search criteria.
    
4. Select the works for which you would like to retrieve citations by clicking the check boxes or the "Select Page" or "Select All" buttons.


5. Click on Create Citation Report at the top right.


6. The H-Index will appear at the right at the bottom of the list.

What are they?

Citation counts are a tempting way to evaluate the importance of an author, article, or journal. It is important to remember, however, as in the case of Andrew Wakefield, that controversial or flawed papers tend to be cited as often or more often that actually valuable and valid research. Especially once the media gets wind of the controversy. It is still useful to know how to do this, though, so here are two simple ways to get citation counts:

Google Scholar

It is (arguably) easiest to do this in Google Scholar. Google Scholar, however, finds the same citing work more than once and counts them all in its total. It's important to go through the list of citing works to remove duplicates rather than to rely on the count that Google Scholar gives.

1. To find an author's works, type in the last name and initials in one of the following formats:
   AJ Wakefield  OR "AJ Wakefield"  OR  author: "AJ Wakefield"  OR  Andrew Wakefield
    
2. Look for the "Cited by" link below each result. This is the number of citations to this work that are indexed by Google Scholar. Click on that link to retrieve them.
    
3. See Google Scholar Search Help for more tips on searching.

Web of Science

Don't like using Google Scholar? You can also use Web of Science:

1. Go to Web of Science. You can use this link if you have it or  from the library homepage, click "Articles & Databases," then "Web of Science" under "Jump to your favorite database."
    
2. Click on the arrow next to Basic Search and change it to Cited Reference Search.
    
3. Enter the author's name, the work's source, and/or the publication year for the work that you would like to see who has cited it.
    
4. Click on "Search" to retrieve the scholarly works that match your search criteria.
    
5. Select the works for which you would like to retrieve citations by clicking the check boxes or the "Select Page" or "Select All" buttons.
    
6. Finally, click on the "Finish Search" button to get your retrieve your results.
    
7. From here, on the top right you can analyze the results or create a citation report based on all of the results.
    
8. For additional assistance, go to "Web of Science: How to do a Cited Reference Search."

Impact factor (IF) "is a measure reflecting the average number of citations to recent articles published in that journal. It is frequently used as a proxy for the relative importance of a journal within its field, with journals with higher impact factors deemed to be more important than those with lower ones. [1]" This type of evaluative tool is linked to citation counts, and both get a lot of criticism. The Lancet, which published the MMR Vaccine study for example has a high IF of 39.207 in 2013. The Lancet is a highly regarded journal and usually has a high impact factor regardless, but this is an important consideration. Here is one way to search for a journal's IF.

1. Go to Journal Citation Reports.
    
2. Depending on the field you would like to search, chose either the JCR Science Edition or JCR Social Sciences Edition.
    
3. To search an edition besides the most recent, use the drop down box to select your preferred year.
    
4. The default view is by journal subject. If you would like to view results by publisher or country, you can click on the drop down box to the right of subject category to select one of these options.
    
5. After you have made your choices, click on the "Submit" button.
    
6. Select your subject area of interest and then use the drop down boxes to select how you would like to view the journal data or aggregate category data.
    
7. If you only want to look at a specific journal or you want to see every journal in the Journal Citation Reports, you can select those options also. Here is an example search:
    
   • Keep JCR Science Edition selected and select Search for a specific journal.
   • Click submit.
   • In the box under enter search term, type in "Journal of Engineering Education" and click search.

• Make sure the file has the correct file name as soon as you upload it
• Authors’ last names (year) title of article or chapter – submitted by _________.
• If you just upload a file, it won't automatically go into the folder where you want it. Be sure to make sure the file is in the correct folder
• Be careful about folder names, especially if you have one folder that you're keeping just for yourself, and another to share articles with your groupmates
• Folders also have icons to indicate whether they are private or shared, to help you keep track of which folder is which

EndNote Online is a reference management tool available online.  While EndNote Desktop is available only to SIU graduate students and faculty, undergraduate students and staff have access to the online version of EndNote.  You can access it and register for an account through the Web of Science database.

The EndNote Online version through Web of Science is available for Windows and Mac users, integrates with MS Word, allows sharing of references with other EndNote online users, and can allow the user to create citations in a variety of styles. 

Users can add references from databases and other sources, store references and attachments, create bibliographies, and Cite While You Write.

EndNote Online also allows you to directly add references from library databases.  The procedure is slightly different for each database. 

Examples of several are below:

EBSCO databases

Web of Science

JSTOR, ProQuest, etc.

Some databases require a few extra steps.  You will need to download an RIS file and import that file into your EndNote online account.

In your search results list, select the items you wish to put into EndNote, click on the "Export Selected Citations" dropdown and select "Export a RIS file"

Save the RIS file to your computer.

Go to your EndNote online account.  Go to "Collect", then "Import References".  Choose the RIS file from your computer, Change the "Import Option" to the database where you collected the reference(s).  Here, we used JSTOR.  Then select the specific group you want to house those references.