How to download a file from a public (no credentials) AWS S3 bucket with python ?

Example of how to download a file from a public, no credentials required here, AWS S3 bucket with python ?

Find an open data project

Let's consider the NOAA public s3 bucket. We want to download a file from the following folder SNPP/VIIRS/SNPP_AF_I-Band_EDR_NRT/2022/09/09/.

Install boto3

To download a file from a aw S3 bucket with python, a solution is to use boto3. To install boto3 with anaconda:

conda install -c anaconda boto3

Find all files in a S3 folder

A first step is often to list all files in a S3 bucket folder. To do that a solution is to do

import boto3

from botocore import UNSIGNED
from botocore.config import Config

s3 = boto3.resource('s3', config=Config(signature_version=UNSIGNED))

bucket = "noaa-jpss"
folder = "SNPP/VIIRS/SNPP_AF_I-Band_EDR_NRT/2022/09/09/"

s3_bucket = s3.Bucket(bucket)

files_in_s3 = [f.key.split(folder + "/")[0] for f in s3_bucket.objects.filter(Prefix=folder).all()]

Note that UNSIGNED has been used here since no credentials are required.

returns here:

  ['SNPP/VIIRS/SNPP_AF_I-Band_EDR_NRT/2022/09/09/AF-Iband_v1r0_npp_s202209082232478_e202209082234120_c202209090027207.nc',
     'SNPP/VIIRS/SNPP_AF_I-Band_EDR_NRT/2022/09/09/AF-Iband_v1r0_npp_s202209082234132_e202209082235374_c202209090027250.nc',
     'SNPP/VIIRS/SNPP_AF_I-Band_EDR_NRT/2022/09/09/AF-Iband_v1r0_npp_s202209082235387_e202209082237028_c202209090027364.nc',
     'SNPP/VIIRS/SNPP_AF_I-Band_EDR_NRT/2022/09/09/AF-Iband_v1r0_npp_s202209082237041_e202209082238282_c202209090025187.nc',
     'SNPP/VIIRS/SNPP_AF_I-Band_EDR_NRT/2022/09/09/AF-Iband_v1r0_npp_s202209082238295_e202209082239536_c202209090025206.nc',
     'SNPP/VIIRS/SNPP_AF_I-Band_EDR_NRT/2022/09/09/AF-Iband_v1r0_npp_s202209082239549_e202209082241190_c202209090027181.nc',

Download a given file

Now to download a file, a solution is to use the boto3 function download_file()

s3 = boto3.client('s3', config=Config(signature_version=UNSIGNED))

bucket_name = "noaa-jpss"
bucket_dir = "SNPP/VIIRS/SNPP_AF_I-Band_EDR_NRT/2022/09/10/"

filename = 'AF-Iband_v1r0_npp_s202209092219338_e202209092220580_c202209100006307.nc'

s3.download_file(Filename=filename,Bucket=bucket_name,Key=bucket_dir + filename)

See also

Links |
------------- | -------------
How to get NOAA VIIRS Active Fires products ? | NOAA Class