Search Prototype

I have been working on a new search interface for GroupServer. The new system
has enough features and enough stability to allow me to demonstrate the
prototype, even though it is far from finished. You can access it from
  http://groupserver.org/groups/development/search_prototype

Currently, only topics are searched.
  * The results are displayed in temporal-order, with the
    most recent item at the top, as is typical for GroupServer.
  * The metadata displayed about each topic should be familiar:
    subject, group, number of posts, and the date of the last
    post. This is the same information as is displayed on the
    Topics page for a group
      http://groupserver.org/groups/development/messages
  * In addition to the standard metadata, six keywords for each
    topic are displayed.

The keywords are extracted using the TF-IDF algorithm. This algorithm states
that a good keyword appears frequently in one document (a topic, in this case)
but does not occur frequently in all other documents. This works surprisingly
well, but it is not perfect. Currently, the keywords are displayed with the
best keyword first, but this may change to alphabetical in the future.

A search may be limited to just a group, which is not the case with the current
system. To do this, click on a disclosure button below the search result, and
click on the link “Only search the group…”. Clicking on “Search for [related]
topics” will display the results of a search that uses the topic-keywords as
the search text.

As is standard with searching, multiple keywords are allowed (separated by
spaces). The results are combined with an inclusive-or: a search for "email
post" will return all topics that contain the word "email", and all the topics
that contain the word "post". This causes quite a few items to be returned when
performing a related-topic search, so I may look ordering the topics by
relevancy in this situation. Only individual words, not phrases, can be
searched, but this will probably change in the future. However, phrase
searching will be *very* slow.

Only the first six topics are returned at the moment, but this will change in
the future. In addition, I will add the ability to view the previous and next
chunks, like the latest topics page, and the latest posts page.

My schedule of items to implement is as follows.
  1. Searching files,
  2. Searching individual posts,
  3. Integration with the rest of GroupServer, and
  4. Searching user-profiles.
The ultimate aim of the integration is for every topic, file, post and
user-list to be generated from the same search code. So the list of topics on
the site and group homepages will be a search, and clicking “View more” will
take you to a search-page.

It is *very* unlikely that GroupServer will support an “advanced search”, which
will allow the use of Boolean operators. Most users cannot write queries with
Boolean operators correctly, and those that can rarely find what they are
looking for when they do. However, I will publish the API, which can be used to
perform weird and wonderful searches that are not supported by the Web
interface.

> Will search result queries result in linkable queries?

I think I understand what you are asking, Steve :)

1. I promise to only use HTTP GET requests for searches. The
   upshot of this is that all searches will be able to be
   bookmarked.
2. The posts and files results will have a link to the Web
   feed for that search. I intend that the user will eventually
   be able to add searches to the site homepage.

Hi, all! I would like to announce the alpha-release of post-searching.
http://groupserver.org/groups/development/search_prototype/index.html?viewTopics=0&viewPosts=1&viewFiles=0&viewProfiles=0

Like topics and files, the posts returned by the searches are ordered by time,
as generally more recent information is the more useful information (just like
news). The data retuned by the search is much the same as you would see on the
latest posts page for a group, such as the one GroupServer Development
  http://groupserver.org/groups/development/messages/posts.html
However, I have also added a snippet from the body of the post. The snippet is
either the first line of the post, or the first line that matches the search
string
http://groupserver.org/groups/development/search_prototype/index.html?searchText=search&viewTopics=0&viewPosts=1&viewFiles=0&viewProfiles=0

Currently the posts are grouped in “chunks” of six, but this will probably be
raised in a subsequent release. What it will be raised to is something that I
am still trying to decide, but for now you can alter the chunk-size by playing
around with the "limit" argument to the search:
http://groupserver.org/groups/development/search_prototype/index.html?searchText=search&viewTopics=0&viewPosts=1&viewFiles=0&viewProfiles=0&limit=12
http://groupserver.org/groups/development/search_prototype/index.html?searchText=search&viewTopics=0&viewPosts=1&viewFiles=0&viewProfiles=0&limit=24
http://groupserver.org/groups/development/search_prototype/index.html?searchText=search&viewTopics=0&viewPosts=1&viewFiles=0&viewProfiles=0&limit=48
http://groupserver.org/groups/development/search_prototype/index.html?searchText=search&viewTopics=0&viewPosts=1&viewFiles=0&viewProfiles=0&limit=96
What would you like it set to?

Like most search engines, post-search supports “phrase searching”, where a
number of words can be grouped together to form a phrase, such as "disabling
delivery"
http://groupserver.org/groups/development/search_prototype/index.html?searchText=%22disabling+delivery%22&viewTopics=0&viewPosts=1&viewFiles=0&viewProfiles=0
Be warned, using phrases slows down the search: you are usually better off
searching for separate words
http://groupserver.org/groups/development/search_prototype/index.html?searchText=unsubscribe+leave&viewTopics=0&viewPosts=1&viewFiles=0&viewProfiles=0

Unlike most searches, we provide a system for altering the parameters of the
search to particular values: parametric search. For posts, the results can be
restricted to a particular group or author. Rather than provide a lists of all
visible groups, I created an interface that allows you to refine the results
based on the posts that have been returned. For example, if you click on
  ▶ Refine
under the last post in the above example, you will see the link
  Show only posts by Michael JasonSmith
Clicking on this will display the following page
http://groupserver.org/groups/development/search_prototype/index.html?searchText=unsubscribe%20leave&authorId=michaeljasonsmith&viewTopics=0&viewPosts=1&viewFiles=0&viewProfiles=0
Which is all the posts that I have written, on GroupServer.Org, that contain
the words "unsubscribe" or "leave". You can perform similar refinement for
posts in groups:
  * All posts in GroupServer Development
http://groupserver.org/groups/development/search_prototype/index.html?groupId=development&viewTopics=0&viewPosts=1&viewFiles=0&viewProfiles=0
  * All posts in GroupServer Development written by Steven Clift
http://groupserver.org/groups/development/search_prototype/index.html?groupId=development&authorId=stevenc&viewTopics=0&viewPosts=1&viewFiles=0&viewProfiles=0
The parametric-search is our greatest departure from the norm, when it comes to
searching, so I would really appreciate feedback on it.

There are a few things that I have not done (yet). I hope to have these done by
the end of the week, unless something gets in my way.
  1. The style of the page is pretty ugly. All the "span" and "div"
     elements are there, with class attributes set, but I have not
     had time to apply the styles to the page.
  2. I do not have an ATOM feed of the post-search. The idea is to
     allow users to subscribe to a Web feed of a particular search,
     such as all posts by me, or all posts in a particular group
     that contain the words "governor".
  3. I have not made a view of the results that uses a table, or
     list, to structure the page.

Future feature enhancements that I am thinking about — but doubt that I will
get around to implementing — are stemming and ordering the posts by something
other than date.

Stemming is the process of determining the root of a word, such as “post” for
“posting” or “posted”. A search that used stemming would return more results
than one that did not. However, it would require a significant amount of work
to add stemming to our system, and we have bigger fish to fry ☺

It would be nice to allow the user to order the results by something other than
time, such as the relevancy score of the post or topic. However, with the
ability to refine the results by author and group, the need to reorder the
results should be reduced.

Enjoy!

As Tim asked so nicely, Web feeds for post searching is now going. There are
still *many* improvements that I need to make, but you will get the basic idea.

> I'm attaching a screen shot from Firefox, in
> case I'm just not seeing it - See Below.

I did send the attachment last time. But, it did not come through.

I'm trying again.

Tim Erickson

Sorry, just a test to see if the attachment goes through correctly for
me.


--Richard

Back to the topic of searching :)

Tim, you are quite right in that the first search page does not have a feed.
  http://groupserver.org/r/file/807746-2007-07-06T225623Z
A feed is only available for post-searches, and the first search-page displays
topics and files, not posts. If you go to a post-search page
http://groupserver.org/groups/development/search_prototype/index.html?authorId=richardw&viewTopics=0&viewPosts=1&viewFiles=0&viewProfiles=0
you should see the link to the feed in the location bar of your browser.

To answer your other question, I would love any feedback you can give me!

The following file was added to this topic:

  feed-indicator.png (16KB)

Richard and I have fixed the dates, so they are formatted nicely, and use the
correct time-zone. While the search-results prompted us to change how we
present dates, all dates should have been altered.

While I have you attention, I would be *very* appreciative if people could test
out the homepage for GroupServer.org, and tell me if the latest-topics list is
visible. I tested the page under Windows XP (IE7, Firefox, Opera, and Safari)
and Linux (Firefox, Opera, Konqueror, and IE6 on Wine) and it all seems to
work, but I have had a report of the page causing an error in Firefox on
Windows.

Thanks for you feedback, Tim! We really appreciate it.

The user of the tags is an infrequent, or first-time, visitor to the site. The
task of such a visitor is to determine what is discussed on a site, or in a
group. The subject-line helps with this task, and the tags reinforce,
supplement, or correct the topic title. What I was aiming to increase the
"information scent" of the page, to take a term from the information foraging
model of Web browsing.

Take the Canterbury Issues Forum as an example
http://forums.e-democracy.org/s/index.html?&g=canterburyissues&t=1&p=0&f=0&r=0
The "Traffic Congestion" topic discusses what? Traffic congestion probably, but
from the tags we also know that public transport is covered, and that the
discussion is talking about congestion in a city, rather than roads being
blocked by cows being moved to a milking shed. What about the "Richard West"
topic? From the topic title we know that it contains posts by, or about,
Richard West, but from the tags we also know that it is also a traffic-related
topic.

Automatically generated tags are not perfect. Like all computing systems they
run by the principal of Garbage In, Garbage Out. The "Access" topic, in the
above example, contains few posts, and none of the tags exemplify the topic.
(It is quite fascinating to read the topic, and try and figure out why "people"
is the top tag.) However, in this example, the metadata from the user
("Access") is also not particularly good either, neither is the title of the
topic "Richard West". Overall, I have found that the tags add value as they
help find information, and give a good impression what is discussed in topics,
groups, and sites.

I do concede that the name "Tag" is misleading. I suggest that we drop the name
altogether, and format the tags in a monospace font, such as Courier. That will
give the impression that the words are from the topic-content, and remove the
source of confusion.

The reason the group name is on a line by itself is to make it easier to scan
the list of topics for groups. A common task is to go the homepage of a site
and establish what is being posted where. Placing the group name at the end of
the topic title makes it harder to scan, as all the group-names are located in
different positions both horizontally and vertically, rather than just
vertically. (It causes layout issues with the size of different parts of the
line.)

I hope this addresses your concerns, Tim :)

As I have received no comments about the prototype file-search, I will roll it
out across the board later today :-)

On Sun, 2007-09-23 at 20:45 -0500, Steven Clift wrote:
> I haven't had time to review in-depth to formulate a response, but was
>  wondering if you compared Google search responses to your own as a
>  test of sorts.

I'm curious as to how that would that work exactly? Google doesn't index
the majority of our groups, or even a good minority of groups with files
to formulate any kind of useful comparison, as far as I can tell.

Google is also incapable of narrowing the search by group or author
(since it doesn't has no ability to determine how those are defined in
our context).

--Richard

Richard Waid wrote:
> On Sun, 2007-09-23 at 20:45 -0500, Steven Clift wrote:
>> I haven't had time to review in-depth to formulate a response, but was
>>  wondering if you compared Google search responses to your own as a
>>  test of sorts.
>
> I'm curious as to how that would that work exactly? Google doesn't index
> the majority of our groups, or even a good minority of groups with files
> to formulate any kind of useful comparison, as far as I can tell.
>
> Google is also incapable of narrowing the search by group or author
> (since it doesn't has no ability to determine how those are defined in
> our context).
>

I am interested in whether our users will find what they are looking for.  My
gut tells me that searches started from a specific group should start with
results just from that group, then have the option to broaden.

Here is one:

http://www.google.com/search?hl=en&as_qdr=all&sa=X&oi=spell&resnum=0&ct=result&cd=1&q=graffiti+site:forums.e-democracy.org&spell=1

From GS:
http://forums.e-democracy.org/s/index.html?s=graffiti

Also, I am often searching months back in our Board group for files/items.
Displaying an option to display results from specific months might be nice.

When it comes to scanning for results I wouldn't mind an option to list more
results. Note how DGroups allows you to see 100 posts and sort by subject and
author:
http://www.dgroups.org/groups/dg-os-study/index.cfm?op=dsp_bydate&listname=dg-os-study&cat_id=2378

Here I can see 30 results quickly:
http://marc.info/?l=zope3-users&w=2&r=1&s=plone&q=b

One idea for navigating older stuff, add a Topics stat line and make the number
clickable to results:
http://groupserver.org/groups/development/stats.html

Cheers,
Steve



>
> --Richard
>
>
> -----------------------------------------
> Full text of this topic in GroupServer Development:
> http://groupserver.org/r/topic/2K38AjS9hjeGC1xar8kpU7
>
> To leave GroupServer Development, email
> <email obscured>?Subject=unsubscribe
>
> GroupServer Development is powered by
> OnlineGroups.Net http://onlinegroups.net
>

On Sun, 2007-09-23 at 21:39 -0500, Steven Clift wrote:
> I am interested in whether our users will find what they are looking
>  for.  My gut tells me that searches started from a specific group
>  should start with results just from that group, then have the option
>  to broaden.

Well, obviously we are interested in that too ;)

I have proposed the same thing, regarding the expectation that users
will have if the search is started within a group.

> Here is one:
>
>
http://www.google.com/search?hl=en&as_qdr=all&sa=X&oi=spell&resnum=0&ct=result&cd=1&q=graffiti+site:forums.e-democracy.org&spell=1
>
> >From GS:
> http://forums.e-democracy.org/s/index.html?s=graffiti

Yes, I see that. To my mind the google search results were much less
useful, given that visitors to a site are typically interested in the
more recent events.

The equivalent post search returns this, BTW:

http://forums.e-democracy.org/s/index.html?s=graffiti&p=1&t=0

I've actually used post and topic searching quite frequently, and it has
typically done a pretty good job.

> Also, I am often searching months back in our Board group for
>  files/items. Displaying an option to display results from specific
>  months might be nice.

See below. I'm far from convinced that specific months is useful, but
time ranges almost certainly are (I have no idea what I would have
talked about in January this year, but I know that we discussed search
sometime in the last 6 months, for example).

> When it comes to scanning for results I wouldn't mind an option to list
>  more results. Note how DGroups allows you to see 100 posts and sort by
>  subject and author:
>
http://www.dgroups.org/groups/dg-os-study/index.cfm?op=dsp_bydate&listname=dg-os-study&cat_id=2378

Yes, that's just an interface change, the feature itself is supported
already.

> Here I can see 30 results quickly:
> http://marc.info/?l=zope3-users&w=2&r=1&s=plone&q=b

30 almost completely irrelevant results ... From a quick flick through
those results, only about 30 actually had *anything* to do with plone.
Note that I actually had to click on every single link to find which
ones were relevant...

I find our results *much* more useful for that use case (searching for a
broad term):

  http://groupserver.org/s/index.html?s=plone

I can immediately see from the keywords of the topics which ones are
most relevant to me, and the posts view of the results gives me even mor
relevance.

I may be completely biased, but I know which one I'd prefer to use from
those two...

> One idea for navigating older stuff, add a Topics stat line and make the
number clickable to results:
> http://groupserver.org/groups/development/stats.html

We don't have any significant 'advanced' search options *yet*. I do
suspect that Mike may have something to contribute when it comes to
temporal ordering, given that his 196 page PhD thesis was entitled "The
Temporal Organisation of Documents and Versions: A User-Centred
Investigation" ;)

Thanks for that Steven, it was a worthwhile exercise. I agree there are
still some tweaks to be made -- to *me* it feels as if we're on the
right track here anyway.

--Richard

The interface Richard and I are talking about is similar to “Sparklines” by
Edward Tufte (Wikipedia came through for me
http://en.wikipedia.org/wiki/Edward_tufte ). It is a cute idea, and could be
quite useful, but I do not see any of us having a chance to implement this
until after GroupServer 1.0 has released.

I have an idea for a barking mad overview+context interface, but that is
something for much later — such as Granita
  http://groupserver.org/groupserver/roadmap/#granita
☺