https://articles.maximemoreillon.com/tags/japanese/ Recent content in Japanese on Maxime Moreillon Hugo -- gohugo.io en-us Wed, 02 Apr 2025 00:00:00 +0000 https://articles.maximemoreillon.com/articles/tokun/ Wed, 02 Apr 2025 00:00:00 +0000 https://articles.maximemoreillon.com/articles/tokun/ <p><a href="https://github.com/maximemoreillon/tokun">tokun</a> is a pet project for learning Japanese from real texts: paste some Japanese in, and the app splits it into words and lets you mark which ones you know. It is a SvelteKit application storing its data in PostgreSQL through Drizzle ORM, and relying on <a href="https://github.com/takuyaa/kuromoji.js">kuromoji.js</a> to do the actual language processing.</p> <h2 id="how-it-works">How it works</h2> <p>Japanese is written without spaces, so the first step is tokenization. When a text is submitted, kuromoji breaks it into tokens, each with its surface form (the word as written), part of speech and reading. The server stores the text, then registers every token. Tokens belong to a user and are deduplicated by surface form, so a word that appears in ten different texts is a single token linked to each text through a join table that also keeps its position. This is what makes the tracking useful: marking a word once applies to every text containing it.</p>