diff options
| author | Pherkel | 2023-09-18 18:13:46 +0200 |
|---|---|---|
| committer | GitHub | 2023-09-18 18:13:46 +0200 |
| commit | f94506764bde3e4d41dc593e9d11aa7330c00e30 (patch) | |
| tree | 6fc438536a72e195805c1aea97926f4c9bbd4f85 /data/tokenizers/tokens_german.txt | |
| parent | 8b3a0b47813733ef67befa6959a4d24f8518b5b7 (diff) | |
| parent | 21a3b1d7cc8544fa0031b8934283382bdfd1d8f1 (diff) | |
Merge pull request #38 from Algo-Boys/decoder
Decoder
Diffstat (limited to 'data/tokenizers/tokens_german.txt')
| -rw-r--r-- | data/tokenizers/tokens_german.txt | 38 |
1 files changed, 38 insertions, 0 deletions
diff --git a/data/tokenizers/tokens_german.txt b/data/tokenizers/tokens_german.txt new file mode 100644 index 0000000..57f2c3a --- /dev/null +++ b/data/tokenizers/tokens_german.txt @@ -0,0 +1,38 @@ +_ +<BLANK> +<UNK> +<SPACE> +a +b +c +d +e +f +g +h +i +j +k +l +m +n +o +p +q +r +s +t +u +v +w +x +y +z +é +à +ä +ö +ß +ü +- +' |