1
$\begingroup$

I am trying to understand how crime frequency affect house price in certain area. To do so, I started with Chicago crime data and zillow real estate data. I want to understand the relation between house price and crime frequency and top 5 crimes in certain areas. Initially, I build up model for this specification, but it wasn't very meaningful to me. Can anyone enlighten me what should I do? any efficient approach to train regression model for potential relation between house price and crime frequency in certain areas? any heuristic idea to move forward?

example data snippet:

here is the merged data that includes annual house price and top crime type in certain areas:

example data

Here is reproducible example data snippet

my attempt

so here is my attempt to fit regression model with above reproducible example data:

from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import StandardScaler
import pandas as pd

regDF = pd.read_csv('exampleDF')

X_feats = regDF.drop(['Avg_Price_2012'], axis=1)
y_label = regDF['Avg_Price_2012'].values

sc_x = StandardScaler()
sc_y = StandardScaler()
X = sc_x.fit_transform(X_feats)
#y= sc_y.fit_transform(y_label)
y = sc_y.fit_transform(y_label .reshape(-1,1)).flatten()
regModel = LinearRegression()
regModel.fit(X, y)
regModel.coef_

but to me, above model wasn't that efficient and needs to be done something more. I think I have to use non linear regression model for those polynomial features, and I am not sure to get this done.

Can anyone point me out how to build correct model for house price prediction over type of crimes and frequencies in certain areas? any idea? Thanks

Goal:

I want to build regression model to predict house price based on crime frequencies and types in certain areas. How can I get modeling the relationship between house price and crimes in certain areas? any thoughts?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

You are probably finding yourself in one of the most interesting problems in data science, the part which is more art than science.

I will give you some ideas that can give you hints on how to solve this problem:

  1. Prices, Salaries and other variables who have information on "accumulables" many times have a distribution which is skewed to the left (many individuals have a little, a few have a lot), what is reccomended to do is to take logarithm of it. Your new variable should be $Ln(Y)$, with this, you will close the gap between areas with greater avg_price and areas with lower avg_price. When that happens you find a less-skewed-normal-like distribution of your $Y$ variable.

  2. The idea of taking Logarithm also applies to the $X$ variables you have (because crimes also accumulate in certain areas).

  3. The Standard Scalling is not necessary when you run a linear regression, because the relativeness of the variable is not influencial in the regression:

The regression $Y = \alpha_0 + \alpha_1X_1+...+\alpha_nX_n$ (no scalled) is mathematically equivalent to $Y = \beta_0 + \beta_1Z_1+...+\beta_nZ_n$ (scalled)

  1. If you want to use other models, you data seems suitable for it, maybe a regression tree or XGBoost might work well on your problem.

I would bet that getting Logarithm in the avg_price, in some exogenous variables and not scalling would get better results for you.

from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import StandardScaler
import pandas as pd

regDF = pd.read_csv('exampleDF')

X_feats = regDF.drop(['Avg_Price_2012'], axis=1)
y_label = regDF['Avg_Price_2012'].values

X = log(X_feats)
y = log(y_label.reshape(-1,1)).flatten()
regModel = LinearRegression()
regModel.fit(X, y)
regModel.coef_
Improve this answer
$\endgroup$
3
  • $\begingroup$ @ Juan Esteban de la Calle can you give me little proof of your point in code? how can I build up meaningful prediction for my data? Thanks $\endgroup$ Commented Apr 24, 2019 at 21:54
  • $\begingroup$ I'll try (in the next answer) $\endgroup$ Commented Apr 25, 2019 at 0:56
  • 1
    $\begingroup$ Regarding point (3), LogisticRegression in sklearn automatically uses regularization, so feature scaling will have an effect. $\endgroup$ Commented Apr 25, 2019 at 1:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.