You can do this from 2 places...
Form the Users > All Users screen, you can select the pencil icon on the right of the user you wish to work with, and in the edit screen for that user, click on the Accessible Budgets menu, and select/deselect the appropriate budgets.
Or you can go to the Budgets > Manage Budgets screen, choose the budget version you need to restrict, and in the Users menu, deselect the users that you don't want to see this version.
You might notice that some of the users in this menu are greyed out and you are unable to deselect them, These are Admin users who have access to everything. If you wish to change their permissions, go to Users > All Users, and in the User Roles section for that user, deselect the top option - Administrator Users have access to all functions. This should allow you to restrict the budgets they can see.