I don't know why this equation have the errors. Thanks if you can help me.
\documentclass{article}
\usepackage{array}
\usepackage{amssymb,amsthm,amsmath}
\begin{document}
\begin{equation*}
\left[h(t)-\lambda t\right]_k=h_{k}-\frac{\lambda}{2 \pi}\int_{0}^{2\pi}e^{i\theta(1-k)}d\theta= \left\{\begin{array}{lcm}
h_{k}-\lambda & if & k=1 \\
\\ h_{k} & if & k \neq 1 \\
\end{array} \right. \\
\end{equation*}
\end{document}
The column type m is wrong. Besides you have a trailing \\ that doesn't belong there.
However, you should use cases.
\documentclass{article}
\usepackage{amssymb,amsthm,amsmath}
\begin{document}
\begin{equation*}
[h(t)-\lambda t]_k
= h_{k}-\frac{\lambda}{2 \pi}\int_{0}^{2\pi}e^{i\theta(1-k)}\,d\theta
= \begin{cases}
h_{k}-\lambda & \text{if $k=1$} \\[2ex]
h_{k} & \text{if $k \neq 1$}
\end{cases}
\end{equation*}
\end{document}
Note that I removed the wrong \left and \right; I added \, in front of the differential symbol.
min array arguments is supposed to mean? If you change withcorlit compiles fineh_{k}-\lambda & if & k=1 \\ \\you've got two\\s there which is probably not what you want?