Hi, I need some help on removing the line noise (50Hz) from my data. I know the method conventionally adopted is to fit a sinusoidal function and subtract from the original data --- the so called notch filter, I have two more questions with this: 1. In my data, not only 50Hz, but the frequencies nearest to it also have abnormal values, this may be caused by the leakage of the power in 50 Hz (I used the simple periodogram), but after I subtract the 50 Hz, it's neighbors are still there. So I am thinking whether the 50 Hz is stable or not (I took only 4 seconds of data). And do have to remove the neighbors? 2. The notch filter seems not very good I think, is there any way to remove this line spectrum while preserving the spectral continuum? Thank you for your attention, any suggestions will be welcome!

# Line noise removal

Started by ●November 28, 2010

Reply by ●November 28, 20102010-11-28

On 11/28/2010 12:06 PM, acat wrote:> Hi, I need some help on removing the line noise (50Hz) from my data. I know > the method conventionally adopted is to fit a sinusoidal function and > subtract from the original data --- the so called notch filter, I have two > more questions with this: > > 1. In my data, not only 50Hz, but the frequencies nearest to it also have > abnormal values, this may be caused by the leakage of the power in 50 Hz (I > used the simple periodogram), but after I subtract the 50 Hz, it's > neighbors are still there. So I am thinking whether the 50 Hz is stable > or not (I took only 4 seconds of data). And do have to remove the > neighbors? > > 2. The notch filter seems not very good I think, is there any way to remove > this line spectrum while preserving the spectral continuum? > > > Thank you for your attention, any suggestions will be welcome! > >As you might imagine: "It depends"..... Subtracting a sinusoid isn't the same thing as a notch filter for a bunch of reasons. - the 50Hz may have sidebands due to frequency modulation because it isn't stable or - the 50Hz may have sidebands due to amplitude modulation because its amplitude isn't stable. Depending on the data, you might use a notch filter of wide enough bandwidth to eliminate all these. It depends on the frequency content of the desired signal. If the 50Hz frequency is stable then you might be able to do something like this: 1) Build a 50Hz bandpass filter with variable gain and with a variable delay at its input or output. This filter will have some total (adjustable) delay at 50Hz. 2) Build a delay element with approximately the same delay as the filter. 3) Feed the signal into both. 4) Subtract the bandpass filter output from the delay element output. 5) Adjust the delay in the bandpass filter path until the delays match and the 50Hz is max reduced. 6) Adjust the gain in the bandpass filter path until the 50Hz is canceled. This is sorta equivalent to an adaptive line canceller except that you're setting the frequency response to be fixed and messing with only the gain and phase/delay. Because you want to cancel out amplitude variations in the 50Hz sinusoid, the delays should be equal. Now, others probably have better methods and this one depends on the character of the "noise" but it could work well enough..... Fred

Reply by ●November 28, 20102010-11-28

On 11/28/2010 12:06 PM, acat wrote:> Hi, I need some help on removing the line noise (50Hz) from my data. I know > the method conventionally adopted is to fit a sinusoidal function and > subtract from the original data --- the so called notch filter, I have two > more questions with this: > > 1. In my data, not only 50Hz, but the frequencies nearest to it also have > abnormal values, this may be caused by the leakage of the power in 50 Hz (I > used the simple periodogram), but after I subtract the 50 Hz, it's > neighbors are still there. So I am thinking whether the 50 Hz is stable > or not (I took only 4 seconds of data). And do have to remove the > neighbors?What is the source of power line noise? Power line frequencies are usually pretty darn stable in most areas of most 1st-world countries. Unless you're taking your data in a remote community or spot that is run off of a small generating plant you should be pretty close to dead-on. How are you sampling the data? Could your data collection have jitter? This would appear as phase jitter in your power line signal. How are you taking the periodogram? If your four seconds of data don't capture an exact integer number of cycles of 50Hz, and if take a FFT, and if you don't window the data, then you'll see spectral leakage as an artifact.> 2. The notch filter seems not very good I think, is there any way to remove > this line spectrum while preserving the spectral continuum?If you are capturing data, then processing it after capture, you can take it's discrete Fourier transform (i.e. FFT), zero out as many bins around 50Hz as you think is appropriate, then take the IFFT. That'll stomp your 50Hz signal dead. It also may well give you some ringing, particularly if you're not windowing things correctly (see my response, above). Making a filter that sneaks up a bit on zeroing out just one bin may be a good idea. HTH. -- Tim Wescott Wescott Design Services http://www.wescottdesign.com Do you need to implement control loops in software? "Applied Control Theory for Embedded Systems" was written for you. See details at http://www.wescottdesign.com/actfes/actfes.html

Reply by ●November 28, 20102010-11-28

>Hi, I need some help on removing the line noise (50Hz) from my data. Iknow>the method conventionally adopted is to fit a sinusoidal function and >subtract from the original data --- the so called notch filter, I havetwo>more questions with this: > >1. In my data, not only 50Hz, but the frequencies nearest to it also have >abnormal values, this may be caused by the leakage of the power in 50 Hz(I>used the simple periodogram), but after I subtract the 50 Hz, it's >neighbors are still there. So I am thinking whether the 50 Hz is stable >or not (I took only 4 seconds of data). And do have to remove the >neighbors? > >2. The notch filter seems not very good I think, is there any way toremove>this line spectrum while preserving the spectral continuum? > > >Thank you for your attention, any suggestions will be welcome!As Fred said, a notch filter is something very different from subtracting the 50Hz. If the 50Hz is mains interference, it should be a narrow spectral line. If the coupling into your signal is stable, the amplitude of the 50Hz interference should also be stable, too. It should be possible to phase lock a 50Hz sine wave to the 50Hz in your signal, and then amplitude lock it to the 50Hz in your signal, and then subtract it from your signal. That kind of approach can work extremely well for a long period of signal, as you can make the bandwidth of the loops very narrow, and get a really good match for the interference. For short bursts of signal you might want to try an interative approach to estimating the 50Hz component. Steve

Reply by ●November 28, 20102010-11-28

On Nov 28, 9:06�pm, "acat" <newthinker@n_o_s_p_a_m.126.com> wrote:> Hi, I need some help on removing the line noise (50Hz) from my data. I know > the method conventionally adopted is to fit a sinusoidal function and > subtract from the original dataNo, it isn't.> --- the so called notch filter, I have two > more questions with this:The notch filter doesn't subtract anything from the data. It attenuates the annoying part of the spectrum.> 1. In my data, not only 50Hz, but the frequencies nearest to it also have > abnormal values, this may be caused by the leakage of the power in 50 Hz (I > used the simple periodogram), but after I subtract the 50 Hz, it's > neighbors � are still there. So I am thinking whether the 50 Hz is stable > or not (I took only 4 seconds of data). And do have to remove the > neighbors?*Subtracting* a complex number from another complex number is the same operation as adding the two complex numbers with an 180 degrees phase difference. For this to work, you need to know two parameters of the spectrum: 1) The magnitude of the annoying component(s) 2) The phase of the annoying component(s) Since you know neither (no, you don't), you actually add two random complex numbers.> 2. The notch filter seems not very good I think, is there any way to remove > this line spectrum while preserving the spectral continuum?No. You can *attenuate* the annoying spectrum components by using a regular filter. Either use a notch filter, which in the ideal world multiplies the magnitude by 0, in which case the 'spectral continuum' is not preserved, or use a narrow-band band-stop filter, in which case the annoying band is not 'removed'. Rune

Reply by ●November 29, 20102010-11-29

Thank you all for the kind replies. (1) I realize the 'notch filter' is indeed not appropriate for what I was doing to the data. I used multitaper method (Thomson) to estimate the 50 Hz noise, the sampling rate was initially 2000 Hz, but I downsampled it to 250 Hz. The estimated amplitude and phase were used to construct the estimate and subtracted from the original data. (2) As Tim Wescott put it, I think the 50 Hz should be stable even the amplitude may drift in my recording. Another idea that came to me is that the clock of my DAQ computer may be exactly the same as the clock of the power plant. So the spectral line may not be on the frequency sampling grid. That will cause some leakage as I just removed the 50 Hz. So I will try remove 50 Hz prior to downsampling. (3) Is there any reference on this topic? (4) I intended to put all the index of annoying frequencies to 0, is this way a little unnatural?>On Nov 28, 9:06=A0pm, "acat" <newthinker@n_o_s_p_a_m.126.com> wrote: >> Hi, I need some help on removing the line noise (50Hz) from my data. Ikn=>ow >> the method conventionally adopted is to fit a sinusoidal function and >> subtract from the original data > >No, it isn't. > >> --- the so called notch filter, I have two >> more questions with this: > >The notch filter doesn't subtract anything from the data. >It attenuates the annoying part of the spectrum. > >> 1. In my data, not only 50Hz, but the frequencies nearest to it alsohave>> abnormal values, this may be caused by the leakage of the power in 50 Hz=>(I >> used the simple periodogram), but after I subtract the 50 Hz, it's >> neighbors =A0 are still there. So I am thinking whether the 50 Hz isstab=>le >> or not (I took only 4 seconds of data). And do have to remove the >> neighbors? > >*Subtracting* a complex number from another complex number >is the same operation as adding the two complex numbers with >an 180 degrees phase difference. > >For this to work, you need to know two parameters of >the spectrum: > >1) The magnitude of the annoying component(s) >2) The phase of the annoying component(s) > >Since you know neither (no, you don't), you actually >add two random complex numbers. > >> 2. The notch filter seems not very good I think, is there any way toremo=>ve >> this line spectrum while preserving the spectral continuum? > >No. > >You can *attenuate* the annoying spectrum components by using >a regular filter. Either use a notch filter, which in the ideal >world multiplies the magnitude by 0, in which case the 'spectral >continuum' is not preserved, or use a narrow-band band-stop >filter, in which case the annoying band is not 'removed'. > >Rune >

Reply by ●November 29, 20102010-11-29

On Nov 28, 8:06�pm, "acat" <newthinker@n_o_s_p_a_m.126.com> wrote:> Hi, I need some help on removing the line noise (50Hz) from my data. I know > the method conventionally adopted is to fit a sinusoidal function and > subtract from the original data --- the so called notch filter, I have two > more questions with this: > > 1. In my data, not only 50Hz, but the frequencies nearest to it also have > abnormal values, this may be caused by the leakage of the power in 50 Hz (I > used the simple periodogram), but after I subtract the 50 Hz, it's > neighbors � are still there. So I am thinking whether the 50 Hz is stable > or not (I took only 4 seconds of data). And do have to remove the > neighbors? > > 2. The notch filter seems not very good I think, is there any way to remove > this line spectrum while preserving the spectral continuum? > > Thank you for your attention, any suggestions will be welcome!You might consider an 'adaptive' filter. This can be done most easily if you have a reference of the 'noise' (in this case your 50 Hz without the signal) that you can use as one input. The filter then adapts to eliminate anything in the signal that is most similar to the noise. Chris ============================== Chris Bore BORES Signal Processing www.bores.com

Reply by ●November 29, 20102010-11-29

On Nov 29, 9:50�am, "acat" <newthinker@n_o_s_p_a_m.126.com> wrote:> (3) Is there any reference on this topic?The literature on DSP.> (4) I intended to put all the index of annoying frequencies to 0, is this > way a little unnatural?No and yes: The *intention* is not unnatural; that's what every newbie and neophyte want to do. The problem is that there is a theorem which is very central to DSP and that I don't remember the name of, off the top of my head, that states that one can not set a continuous frequency band to zero. That will cause all sorts of havoc, e.g. that the system will be impossible to implement. Rune

Reply by ●November 29, 20102010-11-29

On Nov 28, 3:06�pm, "acat" <newthinker@n_o_s_p_a_m.126.com> wrote:> Hi, I need some help on removing the line noise (50Hz) from my data. I know > the method conventionally adopted is to fit a sinusoidal function and > subtract from the original data --- the so called notch filter, I have two > more questions with this: > > 1. In my data, not only 50Hz, but the frequencies nearest to it also have > abnormal values, this may be caused by the leakage of the power in 50 Hz (I > used the simple periodogram), but after I subtract the 50 Hz, it's > neighbors � are still there. So I am thinking whether the 50 Hz is stable > or not (I took only 4 seconds of data). And do have to remove the > neighbors? > > 2. The notch filter seems not very good I think, is there any way to remove > this line spectrum while preserving the spectral continuum? > > Thank you for your attention, any suggestions will be welcome!Besides, what all of the others have said, you also may have harmonics of the 50 Hz, for example 100, 150, 200, 250, etc. Look at the spectrum and see the nature of your interference. You may need several notch filters or even a comb filter. It just depends on your interference. Clay

Reply by ●November 29, 20102010-11-29

On 11/29/2010 02:07 AM, Rune Allnor wrote:> On Nov 29, 9:50 am, "acat"<newthinker@n_o_s_p_a_m.126.com> wrote: > >> (3) Is there any reference on this topic? > > The literature on DSP. > >> (4) I intended to put all the index of annoying frequencies to 0, is this >> way a little unnatural? > > No and yes: The *intention* is not unnatural; that's what > every newbie and neophyte want to do. > > The problem is that there is a theorem which is very central > to DSP and that I don't remember the name of, off the top of > my head, that states that one can not set a continuous > frequency band to zero. That will cause all sorts of havoc, > e.g. that the system will be impossible to implement.In this case, practically, what would happen is that you'd bring the average of the 50Hz signal to zero, but in many places that would leave a lot of "50 Hz-ish" signal there to trouble you. Hence my suggestion to zero out some bins, but to taper the filtering effect. -- Tim Wescott Wescott Design Services http://www.wescottdesign.com Do you need to implement control loops in software? "Applied Control Theory for Embedded Systems" was written for you. See details at http://www.wescottdesign.com/actfes/actfes.html