THE CASE OF THE WEB
ANNELISE MARK PEJTERSEN
AND
RAYA FIDEL
MIRA WORKSHOP
Nancy October 1997
1. Why did we do this work? and how?
Limitations
2. Framework for cognitive work analysis
3. The work context of teaching IR on the web
4. Framework for evaluation of system in work context
5. Evaluation of the web in the teaching work context
6. Suggestions for redesign of the web ?
to analyze searching behavior of high school students when they searched the Web for homework assignments
to suggest changes that may improve the students' learning experience.
Eight high school students, each performing about three searches
Each member of the research team set next to a student while she or he was searching and verbalizing thought processes
Interviews: each student, librarian, teacher and principal
Data:transcriptions of think aloud protocols and interviews
Individual case reports describing searching behavior of each student
Final product in the descriptive part is a description of the students' searching behavior
Why? Priorities and value criteria
What?: General work function
What?: Specific work processes
How? Physical attributes, tools
Survey Question: Draw a map, explain it, ask person to identify the functions she is involved in, redraw the map of what is done, why it is done, what tools are used. Or let person tell about a work case.
Goals and Educate students to be informed citizens, active readers,
Constraints and to have skills in information technology (IT).
Budget limits; state of current technology; state of
technology available in school, e.g., can school's
computers retrieve maps; regulations by the city's schools
system, such as using a certain online catalog system, or
graduation requirements; quality control of teaching
materials; availability of teaching materials to students;
copyright laws; work regulations on city, state or federal
level.
Priority Investment in IT and in teaching materials; integrating IT
into the curriculum; image in the community. Quality
measures: level of library use; level of positive
publicity in media; level of community use of the school's
IT facilities; city and State tests and statistics, such
as demographics, average grade, rate of students'
participation in certain classes; and scores on city,
state and national tests.
General Function Teaching; development of policies; purchase, organized and
make books available; communication with parents;
developing community projects.
Work Process Teaching courses; teaching IT; teaching information
retrieval; teaching subject courses; prepare class
presentations and activities; prepare assignments; grade
assignments; advise students; participate in training
workshops; information retrieval.
Physical Resources Teachers; librarians; staff; teaching materials; library
books; computers; classrooms; library; Internet access in
library; library homepage linked to course material.
Survey Question: What do you do, why do you do it, what tools do you use
Defines the information sources
Teach students: plants outside curricu- lum, what is on the Web on horticulture, to search the www
Priority and quality measures
Integrate www into curriculum
General function
Information retrieval
Work process
Explain IR task, give instructions on navigation and error recovery
Physical resources
Internet, www, computers,networks, library, teacher, librarian
Teacher explained content of assignment in class
Teacher explained how to search
Librarian explained how to search
Students went to the computers in library
Students searched the Web to find answers
Students wrote down answers
Students handed in assignments to teacher
period
Mr. K
Purpose: to learn more about a plant of your choosing. This will expand your knowledge of one specific plant amidst the many plants you are presently studying.
Requirements: -minimum one (written) page
-paper can be computer printed using an eleven or twelve font size
-2 sources properly documented on the last page or separate bibliography page
Discuss in your paper:
1. Genus/species name and common name
2. General description with a sketch or picture
3. origin...where did it come from
4. Historical significance (if possible)
5. Uses (Ex. ornamental, medical, landscaping, food, etc.)
6. Location around the world and in our area (if found here)
7. Growing specifics and the significance of its growing location
8. Personal comments about choice of this plant
Points for Assignment: 25
Learn about one specific plant
Why? Priorities and value criteria
Comments on your choice of plant
What?: General Content
Where did it come from? Location in the world and in our area
Historical significance
What?: Specific content
Growing specifics, genus/species
How? Form, physical attributes
Sketch or picture, Name
period
Mr. K
Horticulture resources
Were Do I Get Information
Directions: The purpose of this assignment is to locate resources you can use to find information about Horticulture topics. The idea is you will find a minimum of 5 resources (excluding...that means NOT including encyclopedias!) and write the following:
--name of resource (book, magazine or other periodical, Internet web site)
--a brief description of what information the resource offers
--how is the resource laid out...diagrams, pictures, step by step instructions to follow, all text (words), etc.
--how you might use the resource in relation to the horticulture class
Here are some web sites to try as well:
http://pathfinder.com/vg http://www.garden.com
http://www.trine.com/GardenNet/AdventGardner
25 points
Locate resources on horticulture- how can resources be used in horticulture class
Not encyclopedias
What?: General Content
What information does the resource contain
What?: Specific content
Horticulture
How? Form, physical attributes
How is the resource laid out,Name and type of resource
Situation analysis; Idea generation; Planning; Evaluate alternatives; compare match
Survey Question: What decisions do you make? What questions do you ask? what information do you need?
Defines information use and queries
Analysis of information needs
Planning a search
Comparison of search results with need (evaluation)
Choose information to write down
Criteria for strategy choice: time, intellectual effort, confidentiality,
reliability, availability
Survey Question: How do you do/what are you looking for?
Defines the content of communication
The Analytical Strategy: Explicit consideration of attributes of the information need, and of knowledge domain.
The Empirical Strategy: Based on previous experience, using rules or tactics that were successful in the past.
Known Site Strategy: Based on understanding the structure of a URL, entering one to retrieve a site.
Similarity Strategy: Find information based on a previous successful example that is similar to the current need.
Examples of criteria:
Time
Mental effort
Fun
Learning new things
Social acceptance among students
High grades
Impressing teacher or librarian
Strategy Time Memory Domain System
Load Knowledge Knowledge
Browsing Much Little Little Little
Analytical Much Much Average Much
Strategy
Empirical Strategy Little Little Little Average
Known Site Little Little Little Little
Strategy
Similarity Little Little Much Much
Strategy
Relates mental models of strategy with levels of expertise/resources and identifies performance criteria; which strategy will be used?
Survey Question: How often do you perform this task? Follow task rules? Depend on your own knowledge? How - use tools?
Defines the form of interface communication
Teacher: Little Web knowledge; depth of domain knowledge unknown
Librarian: Average Web knowledge; little domain knowledge
Rule-based action:
Know-how (procedural knowledge)
Observations
read as signs
Skill-based manipulation:
Sensori-motor operation on objects.
Movements controlled by time-space signals
Skill-rule-knowledge based behaviour
How is it divided, criteria by which people share work ? By organizational tradition, union agreements, work load, functional decoupling, skills, information access, social values and conventions, selforganising interaction and allocation of roles ?
Survey Question: Why do you get this task? Who gives the task? Do you give tasks to others?
Identifies the shared information content for communication on task coordination
Students: searched and gave tips to one another
Teacher: Explained the assignment; gave tips on how to search; and gave help during search about both content of assignment and how to search
Librarian: Gave tips on how to search; gave help during search about both searching and the assignment
Teacher: No role
Librarian: No role
All along, both teacher and librarian gave ad-hoc tips related to all cognitive decision tasks, and so did the students among themselves.
Survey Question: How do you communicate/ cooperate? With whom do you cooperate? Who controls task is done? Is your work important to others?
Determine the form of the commu- nication and coordination of work
Very little financial resources
Very little organization of tasks
Very little organization of cooperation among students, and among teacher, librarian and students
Both teacher and librarian encouraged cooperation of any kind
Spelling and typing "long" URLs
Understanding the searching vocabulary
Reading a homepage at a glance
Understanding the functions of scrolling a homepage
Understanding what the site is about after looking at the homepage
Understanding the meaning of links (or to what they would lead)
Having no information about what strategies were available
Navigational links when they were at the bottom of a homepage and thus were not apparent and easy to miss
Planning a search because there was no representation of search strategies
Understanding the role of the different search engines
having no
information about options in analytical searching (e.g., Boolean, proximity)
Navigation because they could not find out where they were in a search as a whole
Homepages did not reflect the information in the site
There was no information about the subject domain of the site on the homepage
The homepage included no words that were entered as keywords
A homepage did not have graphics
The representation of information on a homepage was cluttered
Difficult to browse when links on the homepage did not relate to the task domain or subject
Getting lost during browsing because of the large amount of links
Problems with Analytical Strategy:
When trying to conceptualize the search within the subject domain Problems with Empirical Strategy:
When a previous tactic did not produce the same results
Not clear if students always recalled tactics and rules that could have helped them
Problems with Known Site Strategy:
When a URL included unrecognized string of characters
When trying to look for a homepage by attributes others than a URL
When looking for a landmark
Problems with Similarity Strategy:
Not supported at all. Strategy Shifts:
Had often problems in applying a strategy and therefore shifted to another strategy
Problems when trying to shift strategies
Difficult to shift from browsing to analytical, or any other strategy, without starting a search from the beginning
Problems with Navigation:
Difficult to figure out where one was, where one is, and where are one's landmarks
Problems in understanding the assignment
Analysis of information needs:
Many problems before and during searching with analyzing information needs. When they were thinking aloud they addressed this issue frequently and also asked for help from librarian and teacher
Problems when trying to retrieve by a specific attribute (e.g., a picture of a plant)
Problems in knowing what type of resources are available in horticulture Planning a search:
There was no planning before the search begun
Problems in having no information about what strategies were available
Many difficulties when planning during the search that generated many requests for help from librarian or teacher Comparison of search results with need (evaluation):
At times there were difficulties in deciding whether a certain sentence on the screen answered a question on the assignment, and students asked librarian for help
Confusion when it was not clear if the results corresponded to the keywords used in searching
Difficulties in comparing a URL on the screen with the one entered
Choose information to write down:
There was no selection of which relevant information to write down
The system did not support the teacher or the librarian in explaining how to search when the assignment was first given
The system did not support the teacher or the librarian when they helped students during the search
Students wrote down sentences from screen, word by word, on assignment sheet, or on a piece of paper. This took a long time.
There was not always enough space on the assignment sheet for the answers
Teacher's goals for the assignments:
Did the Web help students to learn more about plants?
The assignment helped students very little to know what is on the Web on horticulture
Did students learn to search and learn to use the web?
It helped students very little to learn how to find information on the web
The students learned very little about Web searching
There was no communication over the Internet among librarian, teacher, and students
Did the users accept the system in the work situation?
Students accepted and liked searching the web when they had the freedom to decide what to look for, but had great difficulties when they needed to find specific, well-defined information
But Students became disenchanted with the Web for the work context because it took much time and required much mental effort, rarely with complete success.
It was easier to find specific information in a book than on the Web. They had to wait long for graphics to upload
Teacher accepted
Librarian accepted enthusiastically
What impact did the Web have on the work context?
School's goals:
Did the students acquire skills in the use of IT?
IT was integrated into the Horticulture curriculum but the library homepage did not correspond to the course
Circulation of library material tripled
Evaluation touches upon many different facets, and each facet, sub facet, or combination of facets require its own criteria for evaluation
Many suggestions for redesign of the web for this particular work context can be derived from this evaluation
We did not have time to make in depth analysis of users' questions during the IR decision tasks (abstraction levels)
Once you agreed on a how to improve the system, design a study to evaluate its performance.
Your expertise is:
Domain: Web sites Domain: User Needs
Why: Author Intention; Information; Education; Goals: Why ? User's Ultimate Task and Goal
Emotional Experience.
Why: Professional paradigm; Style Literary or Value criteria: Why? Value Criteria Related to
Professional Quality; or School. Reading Process and/or Product
What: General Frame of Content; Cultural General Content: What?
General Topical
Environment, Historical Period, Professional Interest of Historical, Geographical or Social
Context. Setting.
What: Specific, Factual Content. Episodic, Specific content: What? Topical Interest in
Course of Events; Factual Descriptions. Specific Content
How: Level of readability. Physical Accessibility: How? User' s Reading Ability,
Characteristics of sources; Form, Size, Color, perceptual and cognitive capability
Typography, Source, Year of Print.
(Example
of work to be done: analysis of users' questions during IR on the web. Example
from previous work)