DATASETS / KHAZRI CORPUS

An open text corpus for Azerbaijani

Name: Khazri Corpus
Creator: SoftYu Group
License: https://creativecommons.org/licenses/by-nc-nd/4.0/

Khazri Corpus — cleaned Azerbaijani text from more than 570 books and official legal texts, created for NLP research and language model development.

SOFTYU GROUP · AZERBAIJANI / AZ

Khazri Corpus

About the corpus

Khazri Corpus brings together literary, scientific, political and legal writing collected from openly available sources. Texts were cleaned and presented as sentences and short paragraphs. The corpus was used to train Khazri 3.

View on Hugging Face

approximate rows: ~840K
source books: 570+
thematic subsets: 5
text field: text

Inside the corpus

Khazri Corpus — Inside the corpus
Subset	Content and source	Format	Rows
`bedii`	Fiction and literary prose · 300+ books	Parquet	696,848
`elmi`	Scientific and academic writing · 200+ books	Parquet	79,157
`siyaset`	Political writing · 70+ books	Parquet	46,462
`vergi`	Tax Code of the Republic of Azerbaijan	Parquet	2,178
`mulk-mecelle/`	Civil Code of the Republic of Azerbaijan	JSONL	7,701
Total of listed subsets			832,346

The corpus contains approximately 840K rows; the five listed subsets sum to 832,346 rows. The “default” configuration is not counted as a separate thematic subset.

PYTHON / HUGGING FACE

Start working with the corpus

Load a subset with the Hugging Face datasets library, or combine the four Parquet subsets.

pip install datasets

from datasets import load_dataset, concatenate_datasets

bedii = load_dataset("softyugroup/khazri-corpus", "bedii", split="train")
print(bedii[0]["text"])

subsets = ["bedii", "elmi", "siyaset", "vergi"]
corpus = concatenate_datasets([
    load_dataset("softyugroup/khazri-corpus", name, split="train")
    for name in subsets
])

This code combines four Parquet subsets (824,645 rows). The Civil Code is a separate JSONL file in mulk-mecelle/, not a named load_dataset configuration.

Browse JSONL files

A simple data format

Each row has a single text (string) field: a cleaned Azerbaijani sentence or short paragraph. Book metadata and document boundaries are not included.

{"text": "Bu adı Qərbi Azərbaycanda yaşamış orta və yaşlı nəsil həmişə böyük ehtiramla xatırlayır."}

Intended uses

Pre-training and continued training of Azerbaijani language models
Training tokenizers for Azerbaijani
Fine-tuning and domain adaptation
Linguistic research and NLP evaluations

Sources and limitations

Collected from openly available books and official legal texts; no personal user data was collected.
Texts may reflect the views, style and biases of their authors and historical periods.
Author, book title, year and document boundaries are not included.

License

CC BY-NC-ND 4.0

This corpus is released under CC BY-NC-ND 4.0: non-commercial sharing with attribution; redistribution of modified versions is not permitted. Refer to the full license terms when using the dataset.

Citation

@misc{khazri_corpus,
  title        = {Khazri Corpus: A Clean Azerbaijani Text Corpus},
  author       = {SoftYu Group},
  howpublished = {\url{https://huggingface.co/datasets/softyugroup/khazri-corpus}},
  note         = {Hugging Face dataset}
}

Inside the corpus

Khazri Corpus — Inside the corpus
Subset	Content and source	Format	Rows
`bedii`	Fiction and literary prose · 300+ books	Parquet	696,848
`elmi`	Scientific and academic writing · 200+ books	Parquet	79,157
`siyaset`	Political writing · 70+ books	Parquet	46,462
`vergi`	Tax Code of the Republic of Azerbaijan	Parquet	2,178
`mulk-mecelle/`	Civil Code of the Republic of Azerbaijan	JSONL	7,701
Total of listed subsets			832,346

The corpus contains approximately 840K rows; the five listed subsets sum to 832,346 rows. The “default” configuration is not counted as a separate thematic subset.

Start working with the corpus

Load a subset with the Hugging Face datasets library, or combine the four Parquet subsets.

pip install datasets

from datasets import load_dataset, concatenate_datasets bedii = load_dataset("softyugroup/khazri-corpus", "bedii", split="train") print(bedii[0]["text"]) subsets = ["bedii", "elmi", "siyaset", "vergi"] corpus = concatenate_datasets([ load_dataset("softyugroup/khazri-corpus", name, split="train") for name in subsets ])

This code combines four Parquet subsets (824,645 rows). The Civil Code is a separate JSONL file in mulk-mecelle/, not a named load_dataset configuration.

Browse JSONL files

@misc{khazri_corpus, title = {Khazri Corpus: A Clean Azerbaijani Text Corpus}, author = {SoftYu Group}, howpublished = {\url{https://huggingface.co/datasets/softyugroup/khazri-corpus}}, note = {Hugging Face dataset} }