Leaving the controller on during the manual override will not effect anything, simply leave it in the idle state (no need to run any zones or your pump). That said, if you wish to turn it off during this process (just in case), feel free to do so.
As far as the dirt in the valve box, it has likely gotten in from the bottom (as water level rises during the rain, it brings in dirt with it). Alas, any cleanup you will do will likely be only temporary. Ideally valves will rest on the drainage rocks, here is an example (link), as these will be a barrier from water bringing in dirt whenever it gets into the box.
Do keep in mind that dirt should not, in itself, be a reason for concern. Plastic valves should continue working berried as they are. Simply clean up only when it is needed to fix an individual valve. At this point this will be faster / cheaper than redoing your valve box.
Cheers,
Gene