6th June 2013, Adrian Scoica (adrian.scoica@gmail.com)

This corpus was compiled from the German Wikipedia dump from the 9th of March
2013. During the dump processing, only articles with contents larger or equal in
size to 10 KB were kept. This leaves a total of 105820 articles, which are
stored in separate files in groups of 5000 articles per file (with the exception
of one group, which only has 820 articles).

Each of the files representing at most 5000 articles was processed in three
stages:

(A) plain text (*.txt)

  The plain text format contains the raw text content of the articles, after
  stripping away the textile markup. The textile markup was stripped using the
  Nokogiri ruby library, by first creating an HTML representation of the
  document, then removing all <script>, <img>, <table> and <style> nodes from
  it, and then retrieving the remaining text. The resulting output was further
  processed by removing:
    (1) inter-language links
    (2) "[edit]" links
    (3) references
    (4) consecutive empty lines

  The articles were then concatenated, and no further processing was performed
  on the text.

(B) tagged (*.tagged)

  The tagged files were derived from the plain text files, using the
  TreeTagger[0], which was trained on the Negra Corpus[1], and used the
  associated Stuttgard-Tübingen Tagset (STTS), described in detail in[2].

(C) dependency-parsed (*.conll.parsed)

  The parsed files are derived from the tagged text files, using the ParZu
  parser[3].

  The parsed files are in the standard CoNLL format, as output by ParZu.
  The CoNLL format is a tabular format which stores one word per line, with
  sentences separated by a blank line. The syntax of a line describing a word
  is given below:

  WORD-NO WORD-FORM WORD-LEMMA POS-TYPE POS-TAG _ HEAD GR-TYPE _ _

  Where POS-TYPE is a coarse-grained class of the word (eg. noun, verb, adverb),
  while POS-TAG is the full tag. ParZu uses TreeTagger for tagging, and an
  explanation of the POS tags used by the German UTF8 model of TreeTagger is
  given at [2].

  Words are guaranteed to be listed in the same order in which they are found in
  the sentences. For tagged sentences, the HEAD and GR_TYPE fields are blank
  (occupied by a '_' symbol).

Only the final (*.conll.parsed) files are made available in this release.
Please contact the author for accessing the (*.txt) and (*.conll.tagged) files.

The dependency-parsed Wikipedia text is released under the CC-BY SA 3.0 license,
according to the guidelines on[4]. As per the regulations, a copy of the license
can be read here[5].

References

[0] http://www.cis.uni-muenchen.de/~schmid/tools/TreeTagger/
[1] http://www.coli.uni-saarland.de/projects/sfb378/negra-corpus/
[2] http://www.ims.uni-stuttgart.de/projekte/corplex/TagSets/stts-table.html
[3] https://github.com/rsennrich/ParZu
[4] http://en.wikipedia.org/wiki/Wikipedia:Copyrights
[5] http://creativecommons.org/licenses/by-sa/3.0/