| More Information: | Hi Emily,
I enjoyed our discussion over dinner and look forward to collaborating with you
and your students. You mentioned that many of your students are currently
looking for internships; I urge them to consider an internship with the Digital
Discovery group at Amazon.com. Please feel free to distribute this to
interested parties.
The Digital Discovery group is charged with turning the large quantities of
digital content available within Amazon (e.g. text from books in the Search
Inside the book program, or video from the Unbox program) into useful
information for our customers. In addition to handling search queries for
search inside the book internationally, we produce statistically improbable
phrases (SIPs), capitalized phrases (CAPs), reading level and complexity scores,
book-to-book citations and other features. In creating these features, we use
digital content, combined with the rich store of metadata available at Amazon,
from publisher-provided subject categories, to customer reviews to purchase
behavior and sales rank data. There are lots of opportunities to explore data
fusion and augmenting linguistic analysis with external information. We have
plentiful and diverse data and many possible projects to work on. Here are just
a few examples. Any of these projects could
be taken in several directions.
Topic identification/extraction - Determine what set of topics a book covers,
and to what depth. While much work has been done on document classification,
books present an interesting challenge. Many topics can be covered in a single
book, and to different levels of detail. Also, there is interesting work to be
done in determining what the set of topics should be.
Named entity recognition - Extract and classify named entities. We currently
extract capitalized phrases, however this has many limitations. They aren't
classified, and in fact, don't always correspond to actual entities. While
there are many off-the-shelf named-entity recognizers, our corpus is broad
enough to require significant modification/training. Also, we have many
interesting projects for which NER is a jumping-off point.
Citation analysis - Unlike much existing citation analysis that relies heavily
on formatting to recognize formal references, as in scientific papers and some
textbooks, our citation algorithm finds arbitrary mentions of books in prose.
For example, in Edward Eager's "The Time Garden", the following
mention is found, '"This," she said, "is the house where Louisa
Alcott lived. It's the house she wrote about in Little Women. ...' Our
citations algorithm could be significantly improved by using linguistic analyses
to improve precision and recall. We'd also like find citations to other media
(e.g. mentions of movies) and analyze the citation graph.
Stylistic comparison - We currently use our extracted phrases (SIPs and CAPs) to
generate book-to-book recommendations. We'd like to explore other ways of using
content to recommend books. One example is topic-based recommendations
(predicated on topic extraction as described above.) Another is recommending
books by finding authors with similar writing styles.
Machine Reading in books - As described in Etzioni, Banko & Cafarella's 2006
paper, and exemplified in Maiyo, et al.'s paper on Medline question answering,
we would like to mine our large corpus for compelling information and present it
to users in interesting ways. For example, a "who said it" database
that could act as a sort of Bartlett's Famous Quotations for fictional
characters.
Incorporating user feedback - Our customers and authors are a rich potential
source of information, however there are issues to address in using that
information. The general problem of using non-expert input in knowledge
engineering for hybrid systems is one that is likely to be of great value.
If any of these projects sounds interesting to you, or if you have other ideas
of projects that would make use of our Search Inside the Book corpus, please
email me, acarlson at amazon.com.
Thanks,
Adam Carlson |