If you turn Google Assistant off, then it shouldn't be listening for "Hi Google" anymore. When I have it turned off on my phone, it doesn't respond at all regardless of what I say, including "Hi Google."

Whether yours - and everyone else's - cell phones are still listening to everything to say and everything in your environment anyway - is a separate issue.