I have a software filter that multiplies the input signal with a sine and cosine wave (I,Q). After multiply the complex output is square rooted to get the amplitude.
What would be the name of this kind of filter?
What you are describing is not a filter. The multiplication by sine and cosine of frequency f0 is called quadrature conversion. It converts a signal centered at frequency f0 to a complex baseband ("I/Q" or "I + jQ") signal.
To get the amplitude, you take the square root of I^2 + Q^2.
If that is the case then the frequency of the sine and cosine would be irrelevant?
The frequency of the sine wave (aka, local oscillator, in a tuning application) determines which frequency gets mixed (translated) to baseband (DC, zero frequency). That's how a radio tuner functions.
Try looking up "quadrature down-converter" in a textbook or online.
+1 that that's a mixer followed by an amplitude detector. What you're describing is a tuner and a demodulator for AM radio.
Usually that does include a filter, but the process you described didn't include one.
If I understand your description correctly, you are finding the envelop of the signal. kind of Hilbert Transform usecase...