Cloud Upload Example¶
The cloud_upload.py script shows how to upload a file to a Google Cloud Platform (GCP) bucket or an Amazon Web Services (AWS) S3 bucket.
The two functions below are defined in the cloud_upload.py script:
def upload_to_gcp(bucket_name, source_file, destination_file): def upload_to_aws(bucket_name, source_file, destination_file):
They both take the same arguments:
bucket_name = "your-bucket-name" source_file = "local/path/to/file" destination_file = "storage-object-name"
Credentials are stored in a file. For GCP, the user needs to set an environment variable; for AWS, the credentials are hard coded. See below for details.
Two python packages will need to be installed:
python3 -m pip install -r requirements.txt
Each cloud service requires credentials to be properly setup.
For the GCP, the user needs to point to the service account .json file with an environmental variable. Here is an example of how to do that with linux:
Add the above line to your .bashrc file to make permanent. More information (including a Windows example) can be found here.
Running the Example¶
If the cloud credentials have been properly setup, the
main() function in cloud_upload.py is currently configured to upload
requirements.txt to an AWS S3 bucket (simply change
upload_to_gcp to use a GCP bucket instead). A snippet of cloud_upload.py is included below.
aws_bucket_name = 'aws-imagedemo' # s3 bucket name source_file = 'requirements.txt' # sample file to upload destination_file = source_file # using source_file name as destination_file name upload_to_aws(aws_bucket_name, source_file, destination_file)
Run the example script by executing this command in your CLI.
The user should see the below response for a successful AWS upload.
Found credentials in shared credentials file: ~/.aws/credentials (credentials.py:1196) Upload of file requirements.txt as requirements.txt to aws-imagedemo successful (cloud_upload.py:45)
And this reponse for a successful GCP upload.
Upload of file requirements.txt as requirements.txt to c-imagedemo successful (cloud_upload.py:26)