What is an outlier? Explain how you might screen for outliers and what would you do if you found them in your dataset.

What is an outlier? Explain how you might screen for outliers and what would you do if you found them in your dataset. 


Also, explain what an inlier is and how you might screen for them and what would you do if you found them in your dataset.

Outliers:
- An observation point that is distant from other observations
- Can occur by chance in any distribution
- Often, they indicate measurement error or a heavy-tailed distribution
- Measurement error: discard them or use robust statistics
- Heavy-tailed distribution: high skewness, can't use tools assuming a normal distribution
- Three-sigma rules (normally distributed data): 1 in 22 observations will differ by twice the standard deviation from the mean
- Three-sigma rules: 1 in 370 observations will differ by three times the standard deviation from the mean

Three-sigma rules example: in a sample of 1000 observations, the presence of up to 5 observations deviating from the mean by more than three times the standard deviation is within the range of what can be expected, being less than twice the expected number and hence within 1 standard deviation of the expected number (Poisson distribution).

If the nature of the distribution is known a priori, it is possible to see if the number of outliers deviate significantly from what can be expected. For a given cutoff (samples fall beyond the cutoff with probability p), the number of outliers can be approximated with a Poisson distribution with lambda=pn. Example: if one takes a normal distribution with a cutoff 3 standard deviations from the mean, p=0.3% and thus we can approximate the number of samples whose deviation exceed 3 sigmas by a Poisson with lambda=3

Identifying outliers:
- No rigid mathematical method
- Subjective exercise: be careful
- Boxplots
- QQ plots (sample quantiles Vs theoretical quantiles)

Handling outliers:
- Depends on the cause
- Retention: when the underlying model is confidently known
- Regression problems: only exclude points which exhibit a large degree of influence on the estimated coefficients (Cook's distance)

Inlier:
- Observation lying within the general distribution of other observed values
- Doesn't perturb the results but are non-conforming and unusual
- Simple example: observation recorded in the wrong unit (°F instead of °C)

Identifying inliers:
- Mahalanobi's distance
- Used to calculate the distance between two random vectors
- Difference with Euclidean distance: accounts for correlations
- Discard them

Popular posts from this blog

After analyzing the model, your manager has informed that your regression model is suffering from multicollinearity. How would you check if he's true? Without losing any information, can you still build a better model?

After analyzing the model, your manager has informed that your regression model is suffering from multicollinearity. How would you check if he's true? Without losing any information, can you still build a better model? Answer: To check multicollinearity, we can create a correlation matrix to identify & remove variables having correlation above 75% (deciding a threshold is subjective). In addition, we can use calculate VIF (variance inflation factor) to check the presence of multicollinearity. VIF value <= 4 suggests no multicollinearity whereas a value of >= 10 implies serious multicollinearity. Also, we can use tolerance as an indicator of multicollinearity. But, removing correlated variables might lead to loss of information. In order to retain those variables, we can use penalized regression models like ridge or lasso regression. Also, we can add some random noise in correlated variable so that the variables become different from each other. But, adding noise m...
Read more

Is rotation necessary in PCA? If yes, Why? What will happen if you don't rotate the components?

Is rotation necessary in PCA? If yes, Why? What will happen if you don't rotate the components? Answer: Yes, rotation (orthogonal) is necessary because it maximizes the difference between variance captured by the component. This makes the components easier to interpret. Not to forget, that's the motive of doing PCA where, we aim to select fewer components (than features) which can explain the maximum variance in the data set. By doing rotation, the relative location of the components doesn't change, it only changes the actual coordinates of the points. If we don't rotate the components, the effect of PCA will diminish and we'll have to select more number of components to explain variance in the data set.
Read more

What does Latency mean?

What does Latency mean? Latency is a networking term to describe the total time it takes a data packet to travel from one node to another. In other contexts, when a data packet is transmitted and returned back to its source, the total time for the round trip is known as latency. Latency refers to time interval or delay when a system component is waiting for another system component to do something. This duration of time is called latency.
Read more