If the goal is to use the “Clip Browser” audition feature, and drag the related files to your plugin’s window, then yes this could be done fairly easily. Because unlike Regions, the Clip Browser is actually referencing plain-old-files on disk.
It is a credit to Ardour that it’s clip-auditioner is perceived as better than a dedicated drum/sampler tool which one would assume is dedicated for that task.
It occurs to me that Ardour can probably handle a LOT more file formats (.wav .mp3 .ogg .caf etc) than a simple plugin. So even if you can drag/drop it to the sampler, the sampler might not be able to load it. For that reason, I would not assign a very high priority to this feature.
Just to be triply clear: this would only work from the Clip List, not from Regions on the timeline, or the Region List in the sidebar.
-Ben