How to Upload and Download Files with Amazon S3 Using Python
Amazon S3 (Simple Storage Service, or simply S3) is a scalable object storage service from Amazon. It is a reliable and secure cloud solution that enables the storage of unlimited data. In this tutorial, we will show how to upload a file to an S3 bucket. We’ll be using Python along with the Boto3 library for this.
Prerequisites
Before diving into the code, ensure your system meets the following requirements:
- Python and Boto3 installed.
- An AWS account with S3 enabled.
- AWS Access Key ID and Secret Access Key.
Download and Install Python
Even though some of the operating systems are comes with Python as default, however please make sure that you have a python installed on you system. In this tutorial we are using Python3, which you can download form the official website.
Download Page:- https://www.python.org/downloads/
Install Boto3 SDK library
Boto3 is an AWS-provided library that enables interaction with AWS services and account management. You can install Boto3 locally using Python’s package manager, pip.
pip install boto3Now you have the AWS Access key and Secret Key along with boto3 installed on your system, we can jump into write some code.
Upload file to S3 using Python and Boto3
import boto3
from botocore.exceptions import NoCredentialsError, ClientError
# AWS Configuration constants
AWS_REGION = '[your_aws_region]'
AWS_ACCESS_KEY = '[iam_user_access_key]'
AWS_SECRET_KEY = '[iam_user_secret_key]'
S3_BUCKET_NAME = '[s3_bucket_name]'
# Define the file names
LOCAL_FILE = 'upload_file.txt'
S3_OBJECT_NAME = 'upload_file.txt'
def create_s3_client():
"""Create an S3 client using the provided AWS credentials."""
return boto3.client(
service_name='s3',
region_name=AWS_REGION,
aws_access_key_id=AWS_ACCESS_KEY,
aws_secret_access_key=AWS_SECRET_KEY
)
def upload_file_to_s3(s3_client, local_file, bucket_name, s3_object_name):
"""Upload a file to an S3 bucket."""
try:
s3_client.upload_file(local_file, bucket_name, s3_object_name)
print(f'Successfully uploaded {local_file} to {bucket_name} as {s3_object_name}.')
except FileNotFoundError:
print(f'The file {local_file} was not found.')
except NoCredentialsError:
print('AWS credentials not available.')
except ClientError as e:
print(f'Failed to upload file: {e}')
def main():
"""Main entry point of the script."""
print('Starting the upload process...')
s3_client = create_s3_client()
upload_file_to_s3(s3_client, LOCAL_FILE, S3_BUCKET_NAME, S3_OBJECT_NAME)
if __name__ == '__main__':
main()
Here,
S3_BUCKET_NAME: The name of the S3 bucket where files will be uploaded.AWS_REGION: The AWS region where your bucket is located (e.g., ap-south-1, Asia Pacific (Mumbai)).AWS_ACCESS_KEY: Your IAM user’s access key for authentication.AWS_SECRET_KEY: Your IAM user’s secret key for authentication.
Download file from S3 using Python and Boto3
import boto3
from botocore.exceptions import NoCredentialsError, ClientError
# AWS Configuration constants
AWS_REGION = '[your_aws_region]'
AWS_ACCESS_KEY = '[iam_user_access_key]'
AWS_SECRET_KEY = '[iam_user_secret_key]'
S3_BUCKET_NAME = '[s3_bucket_name]'
# Define the file names
S3_OBJECT_NAME = 'path/to/your/s3_file.txt' # The file in S3 you want to download
LOCAL_FILE = 'downloaded_file.txt' # The name for the downloaded file
def create_s3_client():
"""Create an S3 client using the provided AWS credentials."""
return boto3.client(
service_name='s3',
region_name=AWS_REGION,
aws_access_key_id=AWS_ACCESS_KEY,
aws_secret_access_key=AWS_SECRET_KEY
)
def download_file_from_s3(s3_client, bucket_name, s3_object_name, local_file):
"""Download a file from an S3 bucket."""
try:
s3_client.download_file(bucket_name, s3_object_name, local_file)
print(f'Successfully downloaded {s3_object_name} from {bucket_name} to {local_file}.')
except FileNotFoundError:
print(f'The local path {local_file} is not valid.')
except NoCredentialsError:
print('AWS credentials not available.')
except ClientError as e:
print(f'Failed to download file: {e}')
def main():
"""Main entry point of the script."""
print('Starting the download process...')
s3_client = create_s3_client()
download_file_from_s3(s3_client, S3_BUCKET_NAME, S3_OBJECT_NAME, LOCAL_FILE)
if __name__ == '__main__':
main()
Conclusion
In this tutorial, we’ve explored how to upload and download files to and from Amazon S3 using Python and the Boto3 library. To improve the security of your AWS credentials(or any sensitive infromation), it’s best to store them in a .env file or the AWS credentials file at ~/.aws/credentials.