An Envoy should never know more than it needs to.
You control who gets access to your calendar and what they can do with it. Your own Envoy may need broader access to work for you. Someone else's Envoy only needs what's necessary to coordinate with you.
What each Envoy can access
If someone else's Envoy needs your availability
It can only access whether you are free or busy during the requested period. It cannot read your meeting titles, participants, descriptions, locations or any other event details in your calendar.
If it's your own Envoy
It can access the full details of your calendar, including meeting titles, participants, descriptions and other information it needs to manage your meetings on your behalf.
In both cases, you decide who gets access and what they're allowed to do.