Creating multiple KMS keys—and referring to them with unique aliases—is a great way to limit the access to put/get secrets to specific applications or teams.

Instead of using the default alias/credstash alias, you could give a team their own alias and be confident that they aren't going to see or write to anyone else's secrets.