Consider the following guidance to help you choose the right type of AI tool and ensure its outputs are suitable for your needs.
Evaluating AI tools
When evaluating an AI tool for scholarly research, consider:
Does it provide current or live information?
Consider whether the tool is limited by a knowledge cut-off date or can search the web in real time. Many AI tools rely solely on pre-trained data and do not access live information.
Is there a generalist, multidisciplinary or subject specific focus?
Is the tool suitable for establishing a general idea around a chosen topic; does it allow for broad academic inquiry; or is it suitable for a specific field of study?
Is it transparent in its search methodology?
Some tools display their search strategies, demonstrating how they locate literature and arrive at their generated response.
Does it generate responses based on credible scholarly sources?
Unlike scholarly AI databases which rely on academic data, general tools like ChatGPT and Copilot are less likely to produce answers based on peer-reviewed literature.
Does it embrace academic diversity?
Does the tool promote a range of scholarly voices, drawing its response from a variety of sources?
Evaluating AI outputs
Once you have considered the capabilities and limitations of your chosen AI tool, use the following principles to critically evaluate its outputs:
Is the output relevant and accurate?
Check the output against trusted academic sources, especially when using free versions of GenAI tools like ChatGPT and Copilot.
Are the generated sources real or credible?
Always verify whether generated citations have been hallucinated, or fabricated and that the citation's claims can be backed up using credible sources.
Is anything important missing?
AI can oversimplify complex ideas or leave out nuance. Be sure to research further so that you are not missing out on important context.
Would you get a better or different result in another tool?
AI reflects patterns in its training data. If the information seems insufficient, consider using another tool or trusted scholarly source to get the information you require.