9th May 2013, Adrian Scoica (adrian.scoica@gmail.com)

This corpus was compiled from the Spanish Wikipedia dump from the 8th of March
2013. During the dump processing, only articles with contents larger or equal in
size to 10 KB were kept. This leaves a total of 72764 articles, which are
stored in separate files in groups of 5000 articles per file (with the exception
of one group, which only has 2764 articles).

Each of the files representing at most 5000 articles was processed in three
stages:

(A) plain text (*.txt)

  The plain text format contains the raw text content of the articles, after
  stripping away the textile markup. The textile markup was stripped using the
  Nokogiri ruby library, by first creating an HTML representation of the
  document, then removing all <script>, <img>, <table> and <style> nodes from
  it, and then retrieving the remaining text. The resulting output was further
  processed by removing:
    (1) inter-language links
    (2) "[edit]" links
    (3) references
    (4) consecutive empty lines

  The articles were then concatenated, and no further processing was performed
  on the text.

(B) tagged (*.conll.tagged)

  The tagged files were derived from the plain text files, using the FreeLing[0]
  tagger with the included pre-trained Spanish language model, which was then
  converted to the CoNLL format. The CoNLL format is a tabular format which
  stores one word per line, with sentences separated by a blank line. The syntax
  of a line describing a word is given below:

  WORD-NO WORD-FORM WORD-LEMMA POS-TYPE POS-TAG _ HEAD GR-TYPE _ _

  Where POS-TYPE is a coarge-grained class of the word (eg. noun, verb, adverb),
  while POS-TAG is the full tag. An explanation of the POS tags used by the
  Spanish model in FreeLing is given at [1].

  Words are guaranteed to be listed in the same order in which they are found in
  the sentences. For tagged sentences, the HEAD and GR_TYPE fields are blank
  (occupied by a '_' symbol).

(C) dependency parsed (*.conll.parsed)

  The parsed files are derived from the tagged files, using the Malt-1.7.2
  parser[2], with the official IULA-Treebank trained Spanish language model[3].

  The parsed files are in the same format as the tagged files (CoNLL format),
  only the HEAD and GR_TYPE fields are filled-in.

Only the final (*.conll.parsed) files are made available in this release.
Please contact the author for accessing the (*.txt) and (*.conll.tagged) files.

The dependency-parsed Wikipedia text is released under the CC-BY SA 3.0 license,
according to the guidelines on[4]. As per the regulations, a copy of the license
can be read here[5].

References

[0] http://devel.cpl.upc.edu/freeling/
[1] http://nlp.lsi.upc.edu/freeling/doc/tagsets/tagset-es.html
[2] http://www.maltparser.org/
[3] http://www.maltparser.org/mco/mco.html
[4] http://en.wikipedia.org/wiki/Wikipedia:Copyrights
[5] http://creativecommons.org/licenses/by-sa/3.0/