The Re:Search Engine Jaime Teevan MIT, CSAIL Pick a card, any card. Case 1
Case 2 Case 3 Case 4 Case 5 Case 6 Your Card is GONE!
People Forget a Lot Change Blindness http://www.usd.edu/psyc301/ChangeBlindness.htm Change Blindness http://www.usd.edu/psyc301/ChangeBlindness.htm
Re:Search Engine ? Merge Old and New Results Old Merged New
We still need magic! Overview Memorability study
Recognition study Assumptions Implementation issues Evaluation issues Choose your own adventure Memorability Study Participants issued self-selected query After an hour, asked to fill out a survey
129 people remembered something Data Analysis Probability of being remembered Anything? # of words? # of fields? Features Result features: clicked, not clicked, last clicked, rank, dwell time, frequency of visit, etc. Query features: query type, query length, # of
search in session, elapsed time, etc. Remembered rank v. real rank Map remembered rank to real rank Memorability P(Remem|R,C) Clicked - C
Not clicked 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1 2 3 4 5 6
Rank - R 7 8 9 10
Remembered Results Ranked High 12 10 Actual Rank 8 6 4 2 0 -2 -2 0
2 4 6 8 Remembered Rank
10 12 Recognition Study Same set-up as Memorability Study Follow-up survey: Results the same? 16% Case 1: Old results 74% Case 2: New results 65%
Case 3: Clicked to top Case 4: Intelligent merging 17% 92 people have completed both steps Assumptions
Re-search v. search Memorable v. relevant Results change v. stay the same Hide change v. show change Forget v. remember as forgettable Merge v. identify old or new Why?
How to test? What if Im wrong? Implementation Issues Page of cached result may disappear Multiple result pages Identifying repeat queries User identified Search sessions are not repeat queries
Exact query may be forgotten Evaluation Issues Various goals to test Does a merged list look like the original? Does merging make re-finding easier? Is search improved overall? Lab study How to set up re-finding task? Timing differences significant enough?
Longitudinal study What to measure? What are good baselines? Choose Your Own Adventure
Re-search v. search Memorable v. relevant Results change v. stay the same Hide change v. show change Forget v. remember as forgettable Merge v. identify old or new Implementation issues Evaluation issues
Choose Your Own Adventure Re-search v. search Memorable v. relevant Results change v. stay the same
Hide change v. show change Forget v. remember as forgettable Merge v. identify old or new Implementation issues Evaluation issues (Done) Hide Change v. Show Change Why I think change should be hidden
Example: dynamic menus How to prove New results better, called the same or worse Baseline for testing 2 lists, change explicit What if we should show change? Memorability suggests changes to highlight Other applications where want to hide change (Done)
Memorable v. Relevant Why I think memorability is important Relevance at a future date is what matters Necessary to hide change How to prove Baseline for lab study with target first What if relevance is whats important? Mapping between memorable and relevant Useful related work on implicit feedback
(Done) Re-search v. Search Why I think people repeat searches Information seeking literature Re-finding consistently reported as a problem How to prove Study shows prefer to follow known paths Search log analysis
What if people just want to search? Memorable results ranked first Other domains where list consistency matters (Done) Merge v. Identify Old and New Why I think results should be merged Information need not necessarily one or other People dont like to do extra work How to prove
Search log analysis Look at what people do in longitudinal study Lab study timing becomes an issue What if people want to identify query type? Other applications where merging is useful (Done) Results Change v. Stay the Same Why I think results change How search engines work Personalization and dynamic content
How to prove Track query results What if results dont change? Probably will in future applications Existing applications where lists change (Done) Forget v. Remember as Forgettable Why I think people forget
Visual analogy How to prove Lab study Do people find new information? Longitudinal study Ever click on new result? What if remember as forgettable? Build better model of memorability Highlight important changes (Done)
Implementation Issues Page of cached result may disappear Multiple result pages Identifying repeat queries User identified Search sessions are not repeat queries Exact query may be forgotten (Done) Evaluation Issues
Various goals to test Does a merged list look like the original? Does merging make re-finding easier? Is search improved overall? Lab study How to set up re-finding task? Timing differences significant enough? Longitudinal study What to measure? What are good baselines?
(Done) Jaime Teevan [email protected] Strategies for Finding Teleporting Orienteering Why Do People Orienteer?
The tools dont work Easier than saying what you want You know where you are You know what you find Structural Consistency Important
All must be the same to re-find the information! New name Absolute Consistency Unnecessary New name Focus on search result lists
Query Changes Most changes are simple Capitalization Phrasing Word ordering Word form New queries shorter What about longer time horizons? Recognition v. recall
Result List Changes Tracked 10 queries on Google for a year+ 1.18 of top 10 disappear each week Rate of change likely to increase, e.g.: Raw personalization Relevance feedback People forget their queries 28% of queries forgotten within an hour
City of Beaumont. City of Calimesa. City of Canyon Lake. City of Corona. City of Eastvale. City of Hemet. City of Jurupa Valley. City of Lake Elsinore. City of Menifee. City of Moreno Valley. City of Murrieta. City of Norco....
Online Strategy, brand websites, campaigns and CRM enriching for the enablers of our Kit Kat habit. Going from 0 to 100.000+ engaged Facebook fans in 5 weeks for the FIFA FIFPro Player's Union. Creating campaigns for a global phone brand.
Campus Barbers Serve Sincerely. 4th Relation: Propriety. Keep Propriety Citizenry. New Propriety, New Life, New Korea ... Right Driving, At Every Intersection Smiles Bloom. Now We Have to Recycle. ... Traditional Religious Culture in Korea
CNTC S O M M A I R E Présentation du CNTC Accidents Domestiques : Intervention du CNTC. I- GENERALITES SUR LES ACCIDENTS DOMESTIQUES II- ROLE DU CNTC DANS LA PREVENTION DES ACCIDENTS DOMESTIQUES Recommandations et prévention Conclusion. C.N.T.C Forme...
If paying by Travel Card, the payment must be sent with the transmittal memo and the receipt should reflect UCSD as the payee. Reimbursements to the Travel Card through MyTravel must include the transmittal letter and receipt (if available) If...
Any landowner signing either form must be made aware of these facts: Signing any release form is not a requirement of participation in any USDA conservation program and; The person they are authorizing release of the information to, may freely...
Consequences of open framework Compostional Accommodate large cations - Ca, Na, and K Charge balance maintained by exchange of Al and Si Physical properties Specific gravity lower than most other minerals E.g. quartz = 2.65, olivine = 3.27 Not stable...
Circular Holes in Oblique Pictorial Sketches. When circular holes appear in an oblique pictorial sketch, as with isometric sketches, you show the partial edges of the back circle where they are visible through the hole.
Ready to download the document? Go ahead and hit continue!