Using your shared libraries is always a good thing, no? Like your distro’s packages should always have the latest security fixes and such, while flatpaks require a separate upgrade path.
Access to your entire filesystem, however, I agree with you on.
Two questions: are you still on Gentoo, and have you tried LFS?