I am more experienced with flatheads but I'll give ya what I know. Port any rough casting in the head. Leave the intake slightly rough for good fuel atomization, but don't leave enough to disrupt flow. Exhaust should be smooth. Idk if performance cams are available for these vertical shaft singles, but some people weld material to the cam lobes to make them taller, then grind them smooth. I personally don't like this method but if you have the tools to do it and spare cams, it's worth a shot.
I think 20 horsepower is plenty geared right to be honest. You can "adjust" the governor or remove it altogether for more rpms, but over 5-5500 you're in the danger zone.
If anyone has any corrections to anything I have said, please feel free to correct me. I don't want to give false information by mistake