Corpus internal format¶
Single corpus is stored as a directory. In the directory there are several files important for corpus structure.
File config¶
This file stores yaml formatted dict with properties of corpus. A typical config file has following properties:
chunk_size: 52428800
current_chunk: 0
encoding: utf-8
name: 50k Internet Corpus
Warning
you should not modify config file by yourself, unless you really know what you do.
- chunk_size
- max size (in bytes) of single corpus chunk
Note
single document must be stored within single chunk, so you cannot store documents larger that chunk_size.
- current_chunk
- number of current chunk that will be used when appending new document;
Note
chunks are numbered from 0.
- encoding
- internal chunk encoding; possibly always utf-8.
- name
- an optional name for corpus
File chunkN¶
Files like chunk0, chunk1, chunk2, ... contains raw texts and headers. Each chunk can have maximum size of chunk_size bytes config property.
Chunk file has a very simple internal format. Documents are stored sequentially (one after another). Each document is represented as yamled header dict and raw document text encoded with encoding defined in config file.
Internal format of chunk is:
[yamled header1]\n
[raw document1 text encoded]\n
[yamled header2]\n
[raw document2 text encoded]\n
...
[yamled headern]\n
[raw documentn text encoded]\n
Note
chunks are numbered from 0.
Note
single document must be stored within single chunk, so you cannot store documents larger that chunk_size.
An example of two documents long chunk:
id: 8
Prof. Wojciech Roszkowski jest oficjalnym kandydatem AWS na
prezesa Instytutu Pamięci Narodowej - zdecydowało prezydium
Klubu Parlamentarnego Akcji Wyborczej Solidarność.
Rzecznik klubu Piotr Żak przypomniał, że zgodnie z ustawą o IPN,
Sejm wybiera prezesa Instytutu większością 3/5. Do wyboru
Roszkowskiego konieczne jest zatem uzyskanie poparcia nie tylko
Unii Wolności, ale także Polskiego Stronnictwa Ludowego.
Politycy PSL, UW i SLD odmawiają deklaracji, czy ich ugrupowania
poprą kandydaturę prof. Roszkowskiego.
id: 20
Papieże Pius IX i Jan XXIII zostaną beatyfikowani 3 września -
ogłosił Watykan. Beatyfikacja obu papieży zbiegnie się z
uroczystościami Wielkiego Jubileuszu Roku 2000.
File idx¶
- This file contains a list of documents descriptors (indexes in chunk file). This is a list, that contains a tuples like:
- chunk number
- offset of document start in chunk file
- length of header section (with additional \n )
- length of text section (with additional \n)
This file is managed by DB Berkeley Recno structure.
File ridx¶
This files stores a random access index. Basically it is a hashmap containing a mapping of document id to the index in idx list.
This file is managed by DB Berkeley Hashmap structure.