ALTA logo Language Technology Programming Competition 2012
Home
Task Description
Useful Information
Submission
 
 

2012 Shared Task Description

Basic Task Description

July 18, 2012 | Version 1

The goal of this task is to build automatic sentence classifiers that can map the content of biomedical abstracts into a set of pre-defined categories, which are used for Evidence-Based Medicine (EBM). EBM practitioners rely on specific criteria when judging whether a scientific article is relevant to a given question. They generally follow the PICO criterion: Population (P) (i.e., participants in a study); Intervention (I); Comparison (C) (if appropriate); and Outcome (O) (of an Intervention). Variations and extensions of this classification have been proposed, and for this task we will extend PICO by adding the classes Background (B) and Study Design (S); and including sentences that have no relevant content: Other (O). Therefore, the goal will be to classify the provided sentences according to the PIBOSO schema. Such information could be leveraged in various ways: e.g., to improve search performance; to enable structured querying with specific categories; and to aid users in more quickly making judgements against specified PICOSO criteria.

In order to build classifiers, 800 expert-annotated training abstracts will be provided, and the goal will be to build classifiers to annotate 200 test abstracts with the relevant labels. This is a multi-label classification problem, since each sentence can have more than one label. The tagset is defined as follows:

  • Background: Material that informs and may place the current study in perspective, e.g. work that preceded the current; information about disease prevalence; etc;
  • Population: The group of individual persons, objects, or items comprising the study's sample, or from which the sample was taken for statistical measurement;
  • Intervention: The act of interfering with a condition to modify it or with a process to change its course (includes prevention);
  • Outcome: The sentence(s) that best summarise(s) the consequences of an intervention;
  • Study Design: The type of study that is described in the abstract;
  • Other: Any sentence not falling into one of the other categories and presumed to provide little help with clinical decision making, i.e. non-key or irrelevant sentences.

More information about this problem, the construction of the dataset, and a benchmark can be found in Kim et al. (2011)

Data Files and Submission

We will use Kaggle in Class for this year's competition (look for the ALTA-NICTA Challenge). The details about data formats and the submission will be provided in the competition website.

Important Dates

Release of training data On registration
Deadline for submission of results over test data 5 Oct 2012
Notification of results 12 Oct 2012
Deadline for submission of system description poster 28 Oct 2012

 

© ALTA 2012. Competition Organisers.