Thanks, that clarified it a lot. My idea was something like the scala-like specialization annotation that you mention and is also discussed in Primitive Type Specialization - #5 by furunkel. I understand that to implement it as an official language feature, more thought will have to go into it. A fairly naive compiler plugin that generates only specializations by copy&replacing with primitive types as specified by the programmer (and not by the downstream caller) would definitely satisfy my own needs.
Update: Is there a way to mark a post as solution for the thread? I have seen it in other discourse forums but I cannot find it here.