I still experience this issue with A5.12:
Add New Subgroup Bus from group context menu sometimes doesn’t work, and I can’t tell when. It just does nothing…
It works only for groups of audio tracks, and likely also only for mono and stereo tracks.
To expand slightly on Robin’s response, to use that command first you create a group of tracks in the editor and have the tracks assigned to that group, then you right click on a track in that group and create the subgroup bus which should work fine at that point.
It is easy to miss the ‘create the group of tracks’ first step.