Loop through data sets, keep certain variables, save output somewhere else

Question

I'm working on my first Stata project and I have around 30 huge datasets with hundreds of variables. I want a loop that goes through each of them, keeps only the variables I am interested in and saves the reduced form data sets somewhere else. Other solutions didn't work for me 1. because many other people want to overwrite their original sources: for me it is crucial that the original data set remains unmodified and I get the output saved somewhere else to work with. 2. I guess I made a syntax mistake, because I'm relatively new.

Here's what I've produced so far

cd Y:\MAIA\Data\Data2014\

foreach file{
keep nopnltNF NumMois ptwa Qu cvwp ctwpenwp
sort nopnltNF NumMois
save `file'.dta in Y:\MAIA\MyName\test_folder}

Resulting in the error invalid syntax r(198);

foreach file doesn't fit any of the allowed syntax patterns. There is always an in or an of as keyword. Beyond that, did you read in any data before doing this? — Nick Cox
– Nick Cox, Commented Apr 10, 2017 at 12:42
@NickCox No, I had my test file open, but now that you mention it, of course I need to do something so that the loop opens all my data sets one after another. I think this should take care of it, however, it also gives me "invalid" as output... foreach file in wd{ use ´file' . keep nopnltNF NumMois ptwa Qu cvwp ctwpenwp . sort nopnltNF NumMois save `file'.dta in Y:\MAIA\MyName\test_folder} — GaiusBaltar
– GaiusBaltar, Commented Apr 10, 2017 at 13:24
The loop there does no harm but it's just a loop over one file starting with use wd Note the save command is wrong. — Nick Cox
– Nick Cox, Commented Apr 10, 2017 at 13:29

ChrisP · Accepted Answer · 2017-04-14 19:54:06Z

1

Here is one approach to getting a list of Stata files and looping over them.

// get a list of all Stata files in directory
loc datasets : dir . files "*.dta"

// location for transformed files
loc newplace "tmp"

// loop over each dataset
foreach f of local datasets {
    di as result "Reading `f'"
    use "`f'", clear
    keep nopnltNF NumMois ptwa Qu cvwp ctwpenwp
    sort nopnltNF NumMois
    save "`newplace'/`f'"
}

But, note that append can already handle multiple datasets, and has a keep option:

append using `datasets', keep(foo bar)

This will produce one dataset in memory.

answered Apr 14, 2017 at 19:54

ChrisP

5,9721 gold badge36 silver badges36 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Loop through data sets, keep certain variables, save output somewhere else

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related