9th May 2013, Adrian Scoica (adrian.scoica@gmail.com)

This corpus was compiled from the Romanian Wikipedia dump from the 8th of March
2013. During the dump processing, only articles with contents larger or equal in
size to 10 KB were kept. This leaves a total of 9668 articles, which are
stored in two separate files in groups of 5000 and 4668 articles per file,
respectively.

Each of the files representing at most 5000 articles was processed in three
stages:

(A) plain text (*.txt)

  The plain text format contains the raw text content of the articles, after
  stripping away the textile markup. The textile markup was stripped using the
  Nokogiri ruby library, by first creating an HTML representation of the
  document, then removing all <script>, <img>, <table> and <style> nodes from
  it, and then retrieving the remaining text. The resulting output was further
  processed by removing:
    (1) inter-language links
    (2) "[edit]" links
    (3) references
    (4) consecutive empty lines

  The articles were then concatenated, and no further processing was performed
  on the text.

(B) tagged (*.tagged)

  The tagged files were derived from the plain text files, using a
  recently-developed (as of the moment of writing this README) hybrid POS
  tagger[0] which combines statistic tagging models and a rule-based system.
  The tag set used by this tagger is a simplified version of the RACAI tag set
  used in the training data, called the MSD tag set. A detailed description of
  the MSD tag set is given in[1].

  Where POS-TYPE is a coarge-grained class of the word (eg. noun, verb,
  adverb), while POS-TAG is the full tag.

(C) dependency parsed (*.parsed)

  The parsed files are derived from the tagged files, using the Romanian FDG
  Parser, which as of the moment of writing this README is the only publicly
  accessible dependency parser for the language (via a web service[2]). It
  produces output in the XML format, which I then post-processed and converted
  to CONLL format.

Only the final (*.parsed) files are made available in this release.
Please contact the author for accessing the (*.txt) and (*.tagged) files.

The dependency-parsed Wikipedia text is released under the CC-BY SA 3.0 license,
according to the guidelines on[3]. As per the regulations, a copy of the license
can be read here[4].

References

[0] http://nlptools.infoiasi.ro/WebPosRo/docs/Hybrid%20POS%20Tagger.pdf
[1] http://dl.acm.org/citation.cfm?id=1073478
[2] http://nlptools.info.uaic.ro/WebFdgRo/
[3] http://en.wikipedia.org/wiki/Wikipedia:Copyrights
[4] http://creativecommons.org/licenses/by-sa/3.0/