I have a single frequency (100kHz) magnetic wave (known power) signal measured by three, orthogonally sensitive, placed sensors (coils in x,y,z direction). The goal is to estimate the amplitude and direction of the magnetic wave at a certain fixed position in space. We may assume that the orientation of the sensor constellation is known with respect to the magnetic (magnetic moment) direction of the signal generator. There is no modulation, encoding, just a very small band signal.

Currently, we sample each axis independently and correlate a finite set of samples with a cosine and sine (reference signal) at the signal’s known frequency to estimate the amplitude and phase on each axis. Basically it is a dtft at the specific frequency. The phase estimate is with respect to this reference signal.

Since all three sensors measure the same wave (i.e., identical frequency, differing only in amplitude), i’m exploring whether the sampling scheme can be made more efficient by leveraging this redundancy, for example: would an inter-axial sampling offset of 120deg benefit?
Eventually the system should suppress the power at frequencies other than the reference as much as possible. An analog bandpass filter is used prior to sampling. 

I think it is difficult to give a meaningful reply to your question.   What are the orthogonal dimensions of the sensors?   Space?  Time?   Directionality?  Frequency?  Encoding?  It is difficult to assume how your system works with only the description given.

You say you use sin and cos references to estimate the relative phases of the signals, and then say that they're identical in phase and frequency and differ only in amplitude.

Without knowing the nature of the signals, the sensors, or their orthogonality, it is difficult to understand what "an inter-axial offset of 120deg" is or how it might be exploited or not.

Apologies for the lack of detail in my initial message — I’ve updated the original post with more information. If you have a chance, I’d really appreciate it if you could take another look.

Based on the question statement, which seems to be missing a lot of key information, if you have 3 sensors arranged like this in space (all sitting on a common plane in 3D):

   *

*     *

Then you can only accurately measure the time-difference in the arrival of the signal.
Phase differences won't matter as much as time differences for directional estimates. But without knowing the type of signal you are receiving (i.e. wideband vs a constant tone) it's hard to know if measuring phase would be useful or not.

Once you know the time of flight you can estimate a sphere for each of the points to the emitting sound, and look for a common point where those spheres intersect to estimate the position of the emitter.

This will be most accurate in the plane the sensors share.

But I think a more comprehensive problem statement would help us to assist you.

The clarification helps.

I'm guessing that you're careful that your reference signals are in the same phase for each sensor.

Are the sensors spatially separated as well as orthogonally oriented?   I'm having a little trouble figuring out how the phase difference measures direction otherwise.   Is the spatial reception pattern of each sensor known?  

Your DTFT bin will provide a good amount of filtering for adjacent frequency signal rejection, but you could also put some additional filtering ahead of that if needed.    The DTFT bin has a sinx/x frequency response, so if that isn't good enough, you can improve on that with a digital filter.

Unfortunately I still don't know what you mean by "an inter-axial offset of 120deg".   Off the top of my head I don't know of any tricks to apply to exploit the fact that it's the same signal on each axis, anyway.   It needs to be adequately sampled in each axis, and if you're trying to exploit differences in the signal received on each axis you probably don't want to skimp on that.

Anyway, it sounds like an interesting project.

A magnetic 100kHz signal travels through space at the speed of light.

So in order to register an accuracy of 1cm in distance, you'd need a clock running at 29.976 gigahertz and the ability to trigger the sensors and read them at that speed.

If you could do that (which I wager you couldn't), then you could do what you want to do.

But unless you sensors are m's or km's apart, I don't think you'll get much useful information from them.

How far apart are they?

You say that the signals are "identical in phase and frequency and differ only in amplitude", but this isn't actually true, they just look that way because the signal is travelling so fast, at 100kHz, and with any standard sensor, at that speed, time is effectively stationary.

To get any meaningful data, you'd need a MUCH MUCH higher frequency signal, to be able to see ANY difference in phase due to time delay (on the order of GHz minimum).

Light is stupidly fast compared to sound, and measuring it accurately in distance is a nightmare without a dedicated time of flight device.

You're completely right but i'm not using time in this way as I'm not interested in the time difference between the sensor's axes.

The phase I'm referring to is the time difference w.r.t. the reference sine/cosine. Hence, the phase can attain two values (with 180 deg offset), depending in which direction the magnetic wave travels through the coil.

The only place where you state what you are trying to do is: "estimate the amplitude and direction of the magnetic wave at a certain fixed position in space".

And I'm still not sure what this means.

Is the magnetic wave expected to vary at approximately 100kHz also? If now, how fast is it expected to change? What mechanism is changing it?

The direction could only be computed using differences between the sensors, and as I said before, this would be only possible using time of flight, so unless I'm still missing the point, the direction cannot be computed unless you have very high speed time of flight light-speed detectors. Amplitude can be computed using an FFT under the assumption that the arriving magnetic wave is a simple sine or cosine at a fixed frequency. If it isn't, we'd need to know more about what it is.

What is the frequency of the reference sine/cosine? Also 100kHz?

I still think there is a bunch of information missing from the question sorry. Can you supply diagrams to show the arrangement please?

You've also not actually stated what you are trying to do and why you are trying to do it, which i think is crucial to understanding the problem and assisting.

After thinking about this a little more I suspect the phase reverses in the sensor coil depending on the direction of the signal relative to the plane of the coil.   If I have that right it would allow the coils to be colocated and still provide directional information based on the phase.  

I still don't have any ideas on how changing the sampling would improve things, though.  ;)

An interesting project indeed. If I understand your situation correctly you have three mutually orthogonal coils located within a relatively small volume. Each of these coils has a spacial dipole sensitivity so by observing the relative amplitudes and polaritys of the received signals from each of the coils at the frequency you know you are transmitting at you can estimate the direction and distance from the transmitter. Is that more or less correct? 

I worked on a quite similar system (but a totally different application) some years ago. It worked with just three coils. I'm not sure about a forth coil at an other angle but I will think on that. Let me know if I have a correct understanding of the problem.