ILD is not just distance-based attenuation, but also involves "shadowing"-- at higher frequencies (shorter wavelengths) the sound does not refract as well around the head, and will contribute to greater ILD with higher frequency.
The missing dimension is that many sounds have complex spectral characteristics, and the constituent frequencies that make up a complex sound have multiple harmonics.
Imagine a sound directly in front of you, vs directly behind you. The ILD and ITD will be pretty similar.
However, the shape of the cartilage in your ear is not rotationally invariant, it amplifies or attenuates sounds slightly depending both on direction and frequency. So you can have as many directional cues as different detectable frequencies composing your sound, as long as you sort of know the transfer function and the distribution of frequencies in the original sound.
You may want to read more here, week 4 is all about this sound localization problem. https://www.coursera.org/learn/human-brain#syllabus