0

I am working with global applications and decided to use the Python Earth Engine API to collect some data I will need. I am exporting annual, global NDVI composites based on Landsat. Since this involves a lot of data, I split the task in small tiles leading to about 1000 downloads to be conducted. Due to the high numbers of tasks, I run into a 500 error after a while, which is becoming quite annoying.

My question is this: after downloading a file, is there a way to clear the memory used to derive that file?

Improve this question

2 Answers 2

Reset to default
0

There is no need to clear the memory after a task, as this is done automatically in the GEE system.

Exporting a global NDVI composite should be doable without splitting the export into tiles on you own, as this is something that GEE does internally as well.

However, if your batch job is taking too long and you are running into out of memory errors there are a number of ways you can optimize your script - see: https://developers.google.com/earth-engine/debugging#scaling-errors

The most common ones are:

  • calculate geometries only when you need them
  • use bestEffort: true
  • set a higher value for tileScale, such as 16
  • avoid unnecessary type conversions
  • filter first
Improve this answer
3
  • Actually, I also read that GEE should split the output automaticalty. But, after I set a global extent (to download one time-step) it gives me a memory error. I am doing image.getDownloadUrl({'description': oname,'fileNamePrefix': oname,'name': oname,'scale': '300','crs': 'EPSG:4326','region': geometry}) Commented Feb 13, 2019 at 9:10
  • After I get the download link, it usually goes up to >100 downloads before it crashes. But once I re-run my python code, it continues from there. I read that there is a 60 seconds time limit for each download and I seem to be running into it every few hundred downloads. So I was wondering if I can clear the file from the GEE servers after I download it. Commented Feb 13, 2019 at 9:13
  • getDownloadUrl is absolutely not the way to go about calculating a global product. You should use one of the export functions (to drive, gcp, etc.) as they create a batch task that also has more memory available and is allowed to run for longer. Commented Feb 13, 2019 at 9:26
0

Found the problem. The issue was not with my code but with the GEE api. On occasion, it fails to provide access to the file to download but, on a retry, it works. So I added a 'try' and 'except' during the download to retry it until completed. This way I can download a global product in parallel without waiting for GEE to break the file for me (and without crashes).

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.