Karl Palmskog (Dec 08 2020 at 14:37): Karl Palmskog (Dec 08 2020 at 14:37):@Cyril Cohen let's say we converted our MathComp subtokenizer from Python to OCaml. Could this tool be something that lives in the MathComp repo? In a sense, it encodes knowledge about MathComp naming conventions and is the only part of Roosterize that is MathComp-specific.
For benefit of others, some example inputs and outputs for the subtokenizer:
expg1n , [exp; g; 1; n]mulIg, [mul; I; g] Cyril Cohen (Dec 08 2020 at 14:38):I would be nice, but I have trouble understanding which of [exp; g; l; n] and [expg; l; n] is the right tokenization.
Karl Palmskog (Dec 08 2020 at 14:40):well, I guess it is sometimes better to be consistent than to be right, so one would pick one of the them
Last updated: Dec 06 2023 at 13:01 UTC