I learned this weekend that I can use a LLM running on my Macbook Pro to translate notes that I write by hand on a tablet to a markdown text file that I can import in to Obsidian. The benefit is that allows me to securely produce the translation that I can then later use to search for the notes.

My first experience with handwriting on a computer was with the Newton MessagePad, which translated handwriting to text near instantly. The handwriting translation technology in the early 90s was not very good and the Newton’s translation failures earned it ridicule and appearance in the Dilbert comic strip, but I found it did a good job with my fairly neat printing.

Ever since Steve Jobs canceled the Newton upon his return to Apple, there has been no other computing device that defaults to translating handwriting to text. Today most of us who hand write on tablets see their writing stored in its original form, although the graphics is translated to data to be reproduced by the tablet whenever the document opens rather storing the writing in relatively larger bitmap graphics files.

A reason why the Newton translated handwriting to text is that made it possible to search for what was written. Ironically, it seems current tablet makers don’t think it necessary to provide one with ways to search for what they have written. Either the search capability is not provided at all, such is the case of my Viwoods AIPaper Mini, or the on-device search is painfully slow such as on my Boox Note Air 3C.

Microsoft provides an ability to search for handwriting in OneNote on certain platforms. I’ve written hundreds of pages of notes using OneNote on my iPad that that I can search for when the notebooks sync with my notebook computer running OneNote for Windows. It appears that OneNote for iPadOS translates the handwriting and creates a searchable index of the words behinds the scenes but that feature does not exist in the version of OneNote that runs on Android and which I can run on the AIPaper Mini and the Note Air 3C.

Both the AIPaper Mini and the Note Air 3C provide a way to export handwriting in their notebooks to PDF files you can copy to other computers that serve as a simple form of backup, but there is no way to search within those files without having an index generated. I have been on the lookout for any third party tools that I could install on MacOS or Windows that I could use to either convert the “flat” PDFs to searchable forms or at least do a good job of translating the handwriting to text files that I can store in Obsidian.

I recently came across an article written by Jorge Arango titled Transcribe Handwriting Using a Local LLM that describes the steps for performing the translation of handwriting in PDFs to text files using a LLM running on a local computer. If you have personal information in your notes you might not want to send that information to a third party cloud service, which makes this process appealing. This past weekend I tried out the steps in the article on my Macbook Pro M4 with 16 GB of RAM.

My test file was a ten page PDF created on the Viwoods AIPaper Mini that I exported to my Macbook. Arango describes a nearly completely automated workflow of which I manually executed the key steps of splitting the ten page PDF file into ten PNG graphics files and then using the recommended mistral-small3.1 LLM to do the translation into ten markdown files.

The translation of my handwriting to text was pretty accurate but did need some light editing. It took about 70 minutes for the LLM to translate each page. Keep in mind that the VIwoods AI Paper Mini is a smaller, 8-inch tablet so each PDF page is about half the size of a what one considers to be a standard 8.5 x 11-inch sheet of paper that would probably take longer to translate. Arango noted that he uses a 32 GB M2 Max Macbook Pro for the translation and I think RAM was the primary cause of the my longer processing time because the model running under Ollama on my Macbook used 21 GB; more than the 16 GB of RAM it has thus requiring a fair amount of disk swap usage.

The goal of this experiment was to produce a markdown text file copy of my handwriting in a multi-page PDF that I could then add to my Obsidian vault where I could search on it in the future. After I edited each page I used the Unix cat command to produce the single markdown file that I then imported to Obsidian. Finally, I attached the source PDF to the bottom of the file.

I think the benefit of securely translating my writing on my own computer is worth the longer processing time. Now that I have the first ten pages translated, I can go forward by just translating additional pages and concatenating them to the end of the markdown file that I had previously produced. I am certain that if I had used cloud-based version of the LLM that the translation would happen much faster, as no doubt a local computer with more RAM would also perform faster, but I wonder whether the mistral-small3.1 model is the best one for this task and if I could find another LLM with lower RAM requirements that could do as good a job of translation.