Well… its a bit more subtle…
If you are simply recording audio, and playing it back, there is little to be gained by using higher sample rates - the limit of human hearing is 20kHz (and generally gets worse with age). You can completely reproduce the entire audio spectrum by sampling at 44.1kHz.
When it comes to plug-ins, “it depends” - a plug-in might need to run at a higher sample rate internally, if for example it does some form of waveshaping - and the mechanism used for the up / down sampling might be different dependent on the host sample rate, which could account for perceived differences - though you might expect them to be slight. It depends on the design of the plug-in.
upsampling, oversampling or just running everything at a higher rate does not automatically make a plug-in better, in many cases the plug-in’s sample rate conversion to a higher internal rate can add its own artefacts too. Like most of engineering, its a compromise, and different designs / designers choose different trade-offs. It doesn’t automatically mean that the plug-in with the higher sample rate is better.
And be very careful to compare like for like - even a fraction of a dB louder can make things sound subjectively ‘better’, (in fact even a nice fancy GUI can make things sound better…)
Also beware that often obsessing about ‘is the sample rate high enough’ is a distraction from ‘the recording doesn’t sound good, because the recording doesn’t sound good’ - spend more time on capturing the best performance (some of the best analogue recordings still stand up against their modern digital counterparts, and could well have been recorded on (vintage) analogue multi-track equipment with a dynamic range approx. equivalent to 12Bit digital and a top end response below 15K). If your recording sounds bad, its almost certainly not because your using only 24Bit at only 44.1kHz.