Can I remove controller but not lose any info? Shared to diff User

You can transfer ownership to your neighbor. You do this via the same shared access screen you first shared complete access to. Instead of granting complete or limited access, you should see an option for “Transfer Ownership” where you can transfer it while keeping data to your neighbor.