This paper considers panel data regression models with weakly exogenous or endogenous regressors and residuals generated by a multi-factor error structure. In this case, the standard dynamic panel estimators fail to provide consistent estimates of the parameters. We propose a new estimation approach, based on instrumental variables, which retains the traditional attractive features of method of moments estimators. One novelty of our approach is that we introduce new parameters to represent the unobserved covariances between the instruments and the factor component of the residual; these parameters are typically estimable when N is large. Some important estimation and identification issues are studied in detail. The finite-sample performance of the proposed estimators is investigated using simulated data. The results show that the method produces reliable estimates of the parameters over various parametrizations and is robust to large values of the autoregressive parameter and/or the variance of the factor loadings.