Source evaluation

How to evaluate sources, the difference between AI, search engines and academic databases, why bias matters, and why we care about good sources

Validity of information
Bias of information
Making sure we aren't sharing misinformation
Checking that the conclusions we have are accurate
Making sure that the data supports the conclusions shared
Being careful to not fall under propaganda
Example: Climate change
In this day and age anyone can say anything on the internet, it's no longer that info is checked before shared
There is a huge issue with AI giving information that is junk and inaccurate so we need to find where the info is coming from and make sure it's been checked by someone knowledgeable in that arena

Making sure that the data has been analyzed properly by someone that knows what they are doing
Making sure that the data and conclusions were verified by more than one person
Making sure the source is reputable not just AI driven spam or a company trying to get money
For example: some companies publish information that is inaccurate just to sell their products
Some journals and publications are pay to play
Making sure the work was done by experts in the field, not just random keyboard warriors
Harvard on why sources matter

It's only as good as it's sources
It has no checks or oversight, only can compile what was put in
Doesn't understand "relevance" only can parrot back info
Rutgers university on AI and research includes a breakdown some of some AI options and how they could be used

Use an algorithm to decide which links to give you
No checks on accuracy of links or info, you have to decide if you think the link/writer is trustworthy
Santa Clara University Bias in Search Engines And Algorithms

AI technically doesn't learn, it's trained by looking through hundreds and thousands of pieces of information during the training phase
Training is done by giving it info from anything the creators can get their hands on, not always reputable (such as images from social media and art from artists without asking)
The results are just compilation of info and patterns recognized
Example: Lawyer used AI for their arguments and cited fake court cases
When AI Gets It Wrong: Addressing AI Hallucinations and Bias

Search engines go through what's available on the internet and then adds it to its internal list and uses an algorithm to decide what it thinks are the most important
Algorithms have changed over the years, from the simple keyword searches of the 90s, to the more tailored versions of today that take user behavior into account
Search engines are also incorporating more and more AI into both the algorithms and what it shows on page 1, most people don't go past page 1
SEO (Search Engine Optimization) affects the order things come up in, and it changes all the time, it's not based on accuracy of data mostly it's based on how many links link there, or how often people click the link for that search
Search engine algorithms are what decide what's "relevant" or not, but that's not the same thing as accurate.
The 'bias machine': How Google tells you what you want to hear
Example: Ragebaiting or click bait

Should be peer reviewed and checked, not just anyone can post
Community should have already decided if the source and data is reputable
Knowing how to find the info that's relevant is more important, the knowledge has been checked, but it can be hard to find what you want
Should I use library databases to find research instead of Google? Includes Pros and cons

Everyone has an agenda, some are more obvious than others
Everyone has internal biases, even if we don't think we do, we do
Bias can affect everything from where the information is shared to even the words used for sharing
An unfortunate example of this is the passive voice in journalism, and how journalists tend to talk about victims and children
Bias is also an issue when it's a hot topic, because the larger the stakes the more people are inclined to lie or misconstrue results
There are a lot of types of bias, such as confirmation bias, demographic bias, and distance bias
Examples:

Designed by Sarah Blakeslee and their team Original Publication
Evaluating Resources and Misinformation using the CRAAP test includes questions to ask about each source to make sure you're evaluating it thoroughly

Suggested Activities and Discussion Topics:

Activity: Try out a CRAAP test on a news article you've found. Make sure to try finding a news article through a search engine, an AI, and an academic database. Where the CRAAP tests different? Easier to do with one? Different results between them?
Activity: Listen to This Podcast That was created using AI from these materials. Transcript for the Podcast What are your thoughts? Did the AI do a good job representing the materials? Did you find any mistakes?
Activity: Go through This AI generated study guide, what do you think? Did it capture the week materials well? How did you do on the self quiz? Do you know all the vocab used?
Activity: Take a quiz on internal bias such as this one from harvard on implicit associations What did you find? Where you surprised? Is it different then the people around you?
Activity: Watch This AI generated video the Transcript for the Video is here and answer the following questions.

Discussion: Pick an AI such as Gemini or ChatGPT, and request that it generate some content for you based on this material. This could be a summary, a short informational paragraph or an infographic.Now you're going to give it a review based on the following questions:

Did the AI accurately represent the original materials?
Did you find any mistakes, inaccuracies, or "hallucinations"?
Did it capture the essence of the material well, or did it miss nuances?
What does this exercise reveal about the limitations of AI as a primary information source?

Discussion: Find two news articles on the same controversial or "hot topic" from different news outlets.Try to choose sources that might have different perspectives. Answer the following questions

What forms of bias did you see? For example, were there issues with absence of fairness and balance, framing, sourcing, story selection,or tone?
From the article 35 Media Bias Examples for Students, find at least 2 kinds of bias, name them and explain why you drew the conclusions you did, some examples could be "Big story" bias, Corporate bias, Demographic bias, Neutrality bias, Partisan bias, or Confirmation bias.
What internal bias do you have that might contribute to your options?

Would you like to see some more How tos? Click here