Thanks in advance for your help.

It is not easy to see from your plots, but there are clearly some repeating stripes in the lower spectral plot.
16e3 * 0.003 gives 48 samples per frame, but you say that they overlap by 50%, so my guess is that you have some discontinuity in the reconstruction that is giving you a multiple of 333... Hz distortion.

Thanks for your helpful answer. I made a mistake in programming. Now I fix it, but I should consider the critical sampling for these two non-uniform subbands. Since lower band is 1/4 of the bandwidth and higher band is 3/4, and it is impractical to decimate the higher band. However, 'demodulation' maybe help to do this and then decimate by the factor 3/4 as the low band. 

The implementation details of G.722 may give you some insight in terms of split band adpcm coding.

ADPCM is normally applied sample-by-sample in the time domain. I'm not clear on your specific technique, but in general trying to adapt this to a short frame (3 msec) won't be easy without some additional processing.

For debug, my suggestion would be to first implement basic G726 (say 40 kbps) without filtering and make sure the rest of your simulation (I/O, short-time FFT analysis, etc) all looks good. Then add subband filtering and 2x (separate) G726, and again make sure all looks good. Then proceed with your specific modifications.

Feel free to message me privately if there are proprietary details involved.

-Jeff

Thanks a lot for you informative answer. However, I realized that my problem is critical sampling for higher band, in which the Nyquist frequency forbids the decimation, so I have to 'demodulate' the higher band and the decimate it. This is A non-uniform ADPCM , something like ITU, G722 with shorter frame and non-uniform subbands. 

-Niloofar