TREC-IS ran twice in 2019. 2019-B is the second of these. The single task, classifying tweets by information type (high-level) was continued for this edition. Task guidelines are provided and participants could use the 20k examples from the 2018 edition and 10k examples from the 2019-A edition as training data.

You can download the tweet streams from the 2018 and 2019-A edition in conjunction with their assessor labels as training data. See the dataset download page for instructions for the tweets, and the labels can be accessed via the button below.

The metrics summary and evaluation notebooks used to calcuate performance can be found below:

The 2019-B test events are available here and the tweets can be downloaded via the downloader tool using the id 'trecis2019-B-test'.

The full overview paper for 2019 (A and B) will be released in early February. Below you can find the overview slides presented at TREC 2019 along with the plenary intro slides that summarize the planned changes for 2020.

TREC-IS will run twice in 2019. 2019-A is the first of these. The single task, classifying tweets by information type (high-level) is continued for this edition. Task guidelines are provided and participants can use the 20k examples from the 2018 edition as training data.

You can download the tweet streams from the 2018 edition in conjunction with the 2018 assessor labels as training data. See the 2019-A dataset page for download instructions for the tweets, and the labels can be accessed via the button below.

We have updated the evaluation metrics for 2019-A, more information can be found in the document below

Participants should apply their system for a set of 'test' event streams, instructions for accessing these events can be found below:

Your system output can be submitted to the organisers for evaluation. The deadline for submission is June 3rd. Submission instructions can be found below:

A single task is was run for the first year of the track (2018): classifying tweets by information type (high-level). Task guidelines are provided, along with a training dataset where information types and priority levels have been manually annotated for a set of tweets. 'User' profiles are provided for each event type, defining what is relevant for each event type (this is what the assessors are provided when judging).

Instructions for downloading the test dataset, including the topics, ontology and tweets can be found at the link below.

A summary of the evaluation metrics for the 2018 edition is summarized at the following link.

The 2018 Edition of TREC-IS has finished. Across you can find the reports, presentations, publications, open source systems and resources produced from this edition