0

URL return list of files separated by comma example

URL : http://sometimesdownload.com/list_of_files

To get list of files

curl -0 http://sometimesdownload.com/list_of_files

Output of above curl command/List returned :

[" file1.csv","file2.csv","file3.csv"...."filen.csv"]

How to download all files in a single run to a specified path in /tmp/downloaded_files directory in linux and delete the file after downloading. File name of files should be same as original files.

CURL GET to get the files, Download the files and then delete the files after downloading using CURL DELETE.

File can be downloaded like this :

http://sometimesdownload.com/list_of_files/file_name

Example:

curl -o http://sometimesdownload.com/list_of_files/file1.csv

curl -o http://sometimesdownload.com/list_of_files/file2.csv

curl -o http://sometimesdownload.com/list_of_files/file3.csv

Thanks

2
  • What is the meaning of in a single run? Do you mean "using a single curl request"? I don't think that curl is providing this feature. Commented Oct 3, 2022 at 7:13
  • It means executing shell script once... In for loop or any other.. Commented Oct 3, 2022 at 7:43

3 Answers 3

0

You can do this in Java without curl, but if you want to use curl, you're going to want to use either Runnable.exec or ProcessBuilder. See pass multiple parameters to ProcessBuilder with a space for an example

Sign up to request clarification or add additional context in comments.

Comments

0

This could be done as something like

for i in $(curl http://sometimesdownload.com/list_of_files | jq -c '.[]'); then
    # GET file
    curl -O --output-dir /tmp/downloaded_files http://sometimesdownload.com/"$i"

    # DELETE file
    # However this is mostly not working straightforward and need tuning 
    # as no enough information here about how the files are hosted and 
    # configured by the remote server 
    curl -X DELETE http://sometimesdownload.com/"$i"
done

-c is the compact output option in jq to get the plan text output from a json array.

$ echo '["file1.csv","file2.csv","file3.csv"]' | jq -c '.[]'
"file1.csv"
"file2.csv"
"file3.csv"

Comments

0

You can use below script. Since list syntax is different in BASH, you have to convert to bash syntax. [abc,bca] to (abc bca)

And then you can go throw files one by one and download via CURL.

Here is example script, I believe you can correct it according your env.

bashParse() {
    listData=$1
    #validateData
    finalData=()
    echo ${listData} | grep ^"\[" > /dev/null && beginScope=0 || beginScope=1
    echo ${listData} | grep "\]$" > /dev/null && endScope=0   || endScope=1
       
    # build our List BASH stntax    
    if [ $beginScope -eq 0 ] && [ $endScope -eq 0 ] ; then
        # Here I will remove [ and ], also replace last SPACE with nothing => sed
        declare -a finalData=($(echo $listData|tr '\"' ' '| tr -d '[]'| tr ',' ' '))
    fi
    
    # Download files one by one by CURL
    for i in "${finalData[@]}" ; do
       echo "curl://<link>/$i"
    done

}

3 Comments

finalData value : "file1","file2","file3.csv" not working
@kamal-chopra , Sorry. it was my bad. I updated script for printing properly bash Array.
My script is pure bash script. But of course you can use jq tool to parse List and Json.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.