Is it possible to have git track some directory, which is different between two branches, without attempting to merge onto each other branch?

Like we have two branches with the following files:

master:
     src/main.c
     config/main.cfg

testing:
     src/main.c
     config/test.cfg

and I expect every change to main.c would merge on pull. Also both branches would track changes to their respective files in config/, but not showing any changes of the other branch on pull.