Why is visual contact with the conductor important? Let me explain this with a group in a carnival parade.
The Tambour-major (drum major or conductor) leads the group followed by a row of temple blocks (similar to the wood block) and then the Gugge-drums (mobile drum sets), followed by marching bass drums (Timpani) followed by woodwind instruments (often Piccolo), then drummers and finally the brass instruments with the sousaphones usually at the end. A larger Clique (group) has several members and therefore the whole troop can reach a length of 20 meters.
If the persons in the back of the troop (usually the sousaphone players) would synchronize with the sound of the mobile drum set, they would be late by the time the sound needs to travel to them. Assuming the distance between the mobile drum set and the sousaphone is 20 meters, then the time difference is 20 m : 343 m/s = 0.058 seconds (distance divided by the speed of the sound).
This would even be worse for a spectator in front of the group, lets say standing by the temple block: Since the sound from the sousaphone needs also 58 ms to go back to the beginning of the troop, the listener would hear the sound of the sousaphone 116 ms (2 * 58 ms) later than the sound of the drum set. A sixteenth note played at a tempo of 120 beats per minute, and the beat is set to be a quarter note, has a duration of 125 ms (60 s : 120 : 4). So for 16th notes, two sounds, - the sound produced where you stand e. g. the temple block and the sound from the sousaphone player -, could come together, which probably were not meant to meet (e. g. a C and a C sharp played together is not a pleasing sound, i.e. not harmonic).
If the sousaphone player follows the conductor the time difference is only half the time: 58 ms. Since the time for the synchronizing signal from the conductor occurs with light speed, the delay can be ignored. Which is significant, since the human brain can distinguish events more than 100 ms apart easily, but below that, depending on the events, it gets difficult.
This is one reason why an orchestra needs a conductor: To be in synchronization with each other.
To summarize: Despite the delay, the music sounds good, because our brain automatically corrects these timing errors. It is like listening to someone, which swallows the endings of words. The redundancy in the spoken word let us recover the meaning of the sentence, without even realizing that we are correcting the words. This auto correction is also true for music. In this way our brain deceives us from a scientific perception what is really going on. But this is to our own advantage: The whole sound picture falls into a pleasant and meaningful experience. Without auto correction, the perception of the world would be too complex (every music piece you hear has deviations).
To be a good conductor, you have to train your brain to stop this auto correction and start to differentiate accurately. If you are not ready for that, your brain tells you that he is overburdened with this task. Luckily, your brain also tells you to use tools, if you are not capable of performing a particular task. For example you need a microscope in biology to observe details of small insects, your eyes are not made for such a task.
A tool, which helps musicians to get accurate feedback, is Listening Singing Teacher. Listening Singing Teacher gives you feedback on your timing when doing rhythm exercises. In this way you can see, if you have a tendency to be late or early.
Sample feedback from Listening-Singing-Teacher:
Listening Singing Teacher, Listening Music Teacher, Listening Ear Trainer, The Red Pitch Dot, The Colored Pitch Line, The Counting Hints Line, The Half-Step Brackets, The Precision Listening Method, The Singing Funnel Method, The Octave Anchor Pitches Method,The Interval Overtone Method, The Pitch Keeper Method, Absolute Pitch Point and Same Pitch Please are trademarks of AlgorithmsAndDataStructures, F. Rudin. Macintosh and OS X are trademarks of Apple Computer Inc., IBM PC is trademark of International Business Machines Inc., Windows XP/Vista/7 is trademark of Microsoft Inc. All other company and product names are trademarks or registered trademarks of their respective owners