Certainly we must be aware of dangers and plan accordingly, but do not take counsel of your fears. Your fears may be correct - but they may well be wrong.
You must not give up nor give in. If you try you may fail, but if you don't try you will certainly fail.
If those sound like platitudes - well, there is a reason why some advice is repeated often: because it is true, and because it is good.
Otherwise, all I can do here is to express my support for you and for your relationship, and to let you know you are not alone.