Narrowband filtering for short signals

TLDR: For short signals, is extracting a narrowband signal with FIR/IIR filters a fool's errand? Specs: signal = 936 samples long, target sinusoid is 50.05 samples long.
I’m working with a sinusoidally stimulated physical system whose response is measured by a sensor. My goal is to measure or estimate the magnitude of the system's sinusoidal response. Here are the specifics:
- Signal Characteristics:
- The stimulus frequency is continuously varying, but I’ve remapped the measured signal from magnitude-vs-frequency to position-vs-frequency. This results in a sinusoidal signal.
- The measured signal also contains unrelated components: vibrations, resonances, and noise from other physical effects. However, there are no system resonances within my frequency range of interest (17.7–19.7 periods).
- Filter Design Attempt:
- I initially tried a Butterworth bandpass filter to isolate the sinusoid of interest and then fit a sinusoid to the filtered signal. However, the filter's impulse response appears too long, given my short signal.
- The signal is 936 samples long, with the period of the sinusoid of interest being 50.05 samples (corresponding to ~18.7 rotations over the signal duration).
- Filter Issues:
- The narrowband nature of the filter (bandwidth ~2 Hz) introduces a long impulse response. Is this the same as saying the filter has a long group delay, and if so, is this inherently problematic for short signals like mine?
- If filtering is possible, is there a better filter design (e.g., FIR or different IIR topology) that would work here?
- Alternative Approach:
- I’m now considering convolution with orthogonal sinusoids (e.g., multiplying the signal by
cos(2πf₀t)
andsin(2πf₀t)
, then summing over time to compute the magnitude). - Does this sound like a better approach than trying to filter the signal? While I expect this to work, it would be valuable for intuition (and communication with others) to visually show the extracted sinusoidal signal.
- I’m now considering convolution with orthogonal sinusoids (e.g., multiplying the signal by
- FFT Consideration:
- Since the sinusoid of interest spans ~18.7 cycles (not a whole number), I assume FFT filtering will also fail to cleanly isolate it. Am I correct?
Key Question: For signals of this length and frequency characteristics, is narrowband filtering inherently problematic, and would convolution-based extraction (or another method) be more appropriate? Any insight into improving filter designs for this kind of scenario would also be greatly appreciated.

Convolution with orthogonal sinusoids IS effectively the FFT (which to be clear is correlation with orthogonal sinusoids). The limited time capture results in a wider resolution bandwidth which then limits the amount of achievable noise filtering. If you knew that the noise sources were white- then a boxcar average would be the optimum strategy which is doing an FFT with no windowing. In most cases we can’t count on having other interference present, so windowing is necessary to reduce the errors due to spectral leakage. Zero pad the waveform prior to taking the FFT (out to 4 to 10x the total number of samples) to eliminate scalloping effects. Scale the FFT by the sum of the window used and you will get the best estimate for the tone amplitude (you’ll actually get the estimate of the positive and negative frequency components as two tones for a real sinusoid). This method is very accurate and as you are finding will be limited by the total capture - however this is with that windowed FFT an optimum correlation with reference sinusoids (actually reference e^(j omega t)

If this is an offline post-processing problem (say in MATLAB), you can use the least-square based method below which is an old (but perhaps not very well known) trick I use often to calculate SNR / THD+N of single tone signals.
For such a tonal signal, if the frequency w is known, you can assume that your signal can be reasonably modeled as
y(t) = c1*cos(wt) + c2*sin(wt) + residual_noise
To find c1 and c2 you can formulate this as a linear least square problem A*x = b and solve it with MATLAB's \ operator (numpy.linalg.lstsq in python). Here b is your N by 1 signal, x is a 2x1 matrix [c1; c2], and A is a N*2 matrix [cos(wt); sin(wt)].
This method gives you the exact amplitude at that frequency without having to worry about DFT leakage effects. The least square solution by definition is a tone with frequency w and amplitude of sqrt(c1^2+ c2^2) that minimizes the residual noise power. However, for this to work the frequency has to be known EXACTLY.
If the frequency is not known, you can first use an FFT to find the approximate tone frequency (highest bin), then iteratively solve the least-square problem with an optimization algorithm where the optimization variable is the frequency and the cost function to be minimized is power of the 'residual_noise' i.e. sum(residual_noise.^2) from above. MATLAB's fminunc works well. This cost function converges very well because the closer the assumed frequency is to the actual tone frequency, the lower then residual noise power.
Of course, this is a lot of math to do online, but in MATLAB it would be almost instant given how short your signal is. And in my experience it is extremely accurate in finding both the tone frequency and amplitude.

You don't say exactly what you're trying to extract from the signal, so it's a little difficult to answer in details, but I can give some general responses.
In general a narrow filter has a long impulse response, so, yes, it can be very difficult to extract or isolate details in the frequency domain if the signal duration is short. One way you can think of this is that the time-domain signal has had a rectangular window applied to it, and as that rectangular window gets shorter, the frequency domain sinx/x response gets wider. This is sometimes referred to as a time-frequency uncertainty principle, where increasing resolution in one domain decreases it in the other. So it just comes with the territory, but the behavior is predictable so you can plan for it and try to work around it for whatever specific information you're trying to get out of the signal.
Depending on what you're trying to isolate in the signal, there are a number of tricks that you can try that might be useful.
1. If you want a narrow filter response, you can design an IIR filter with the characteristic that you want, and then pass the signal through the filter, then reverse the order of the signal and pass it through again (or vice versa, either way). This results in no group delay distortion because the group delay has been applied in both directions. The filter response actually be the square of the IIR filter response, so stopband attenuation will be doubled, etc. Usually this is done by zero-padding both ends of the signal and then figuring out where to chop the tails off to get the original samples back.
2. FFT bins are all the same, so the low frequency bins, or even the DC bin, have exactly the same frequency response as the others, so if any bin or bins provide the frequency resolution that you need, then there's no reason not to use an FFT.
3. If you want narrower FFT bins, you can zero-pad the time domain signal and use a longer FFT. The bins will be narrower, but you won't really increase the frequency resolution due to the original finite length of the signal (as previously mentioned), but it will be more oversampled in the frequency domain if that's useful to you.
All of these things come with a lot of caveats and limitations and pitfalls, but you learn those sorts of things by doing things like this.
Best of luck with it!

If your signal of interest is 18 or so cycles, can't you just detect it in time domain instead of going to frequency domain filtering.
In time domain you can use correlation for example with a known reference frequency.

Can the problem be treated as estimating a sinusoidal signal within a "noisy" measurement? The sinusoid can be modeled perfectly as a second-order differential equation. If the noise were white, this can be solved optimally with a Kalman Filter. If you send along a dataset, I'll process with the KF. Also need the delta-time for the samples and the frequency (assumed known?) of the sinusoid. The solution will provide the estimation of A & B for: s(t)=A Cos(wt) + B Sin(wt)
The key to the degree of success will be the effective signal/noise as can be determined by inspecting the residuals from the KF.
Jim Kain

For very short captures (even for less than one sinewave period), a remarkably effective method for sinewave parameter estimation (amplitude, frequency, phase) is curve fitting.
I wrote a short example here back in 2014: