Private files are accessible through URL regardless of taxonomy access.

I have a site with a multi tenant situation including different roles for each tenant. I'm managing access to content through taxonomy terms and views, which works well so far. However, I have different file types that should be accessible to different roles within one tenant. Content or files of one tenant should never be visible to anyone (user or role) of another tenant.

I have only found the global access permission "view media". When that is not checked for my role, I can't access the file even though I see it in the view (which makes sense). However, when that is checked for the role, the role has access to any file (including other tenants) through the url (even if they don't have access to the view that would link to the file).

Is there a module that can differentiate view media permissions by media type? My research only showed small, instable or experimental modules.
Or do you have any other suggestions on how I could solve this issue?