Apyfal storage provides the ability to use cloud storage services and other storage as a source and target for an accelerator.
Using this feature to handle files provides some advantages:
- When using Apyfal to remotely control an accelerator, all file transfers are performed on the host directly.
- Apyfal storage uses simple URLs like
strto define files.
- Apyfal storage can also be used to handle basic copy or open operations on storage services files.
Apyfal storage URL format¶
Apyfal storage works with extended URLs that support extra schemes.
Apyfal use the standard URL format
Apyfal storage supports the following basic schemes.
file: Local file on file system (
filescheme is assumed if no scheme provided). Example:
https: File available on HTTP/HTTPS. Example:
Cloud storage schemes¶
Apyfal storage supports extra schemes for cloud storage services. In this case, the scheme is the service name or the provider name.
Cloud storage use buckets (or containers) to hold data. The bucket name needs to be specified just before the file path in URL.
See apyfal.storage for information available storage services and the scheme to use.
s3://my_bucket/my_file: File with
my_filekey on AWS S3
ovh://my_container/my_file: File with
my_filename on OVH Object Store
host scheme is similar to the
file scheme, but is only available
when using Apyfal to control the accelerator remotely.
In this case:
filerepresents a client-side file that needs to be transferred to/from host.
hostrepresents a host-side file that can be used directly.
For security reason,
host scheme is restricted to a whitelisted list of
directories on host.
This list can be modified, host side, using the
security section in the configuration file.
The only default authorized directory is
Using storage with Accelerator¶
apyfal.Accelerator has native Apyfal storage URL support for file
import apyfal with apyfal.Accelerator(accelerator='my_accelerator') as myaccel: myaccel.start(src='my_storage://src') myaccel.process(src='my_storage://src', dst='my_storage://dst')
Basic storage operations¶
Apyfal storage provides some basic file operation functions to easily manipulate storage files:
apyfal.storage.open: Open a file as file-like object. Like builtin
apyfal.storage.copy: Copy a file between two URL. Like
The following example shows some possible file operations:
import apyfal.storage # Open file as text for reading with apyfal.storage.open('my_storage://my_file', 'rt') as file: text = file.read() # Open file as binary for writing with apyfal.storage.open('my_storage://my_file', 'wb') as file: file.write(b'binary_data') # Copy file from storage to local file system copy('my_storage://my_file', 'my_file') # Copy file from local file system to storage copy('my_file', 'my_storage://my_file') # Copy file between storage copy('my_storage://my_file', 'my_other_storage://my_file') # Download a file from internet to storage copy('http://www.accelize.com/file', 'my_storage://my_file')
Mount extra storage services¶
Cloud storage services use a login and password to secure access and can’t be accessed without them.
By default, storage services that are already configured as host are automatically mounted with same parameters.
But, in other cases, these services need to be mounted before use.
Each storage needs a unique
storage_type that will be used to mount it
and to access it with a URL.
This can be done using the
apyfal.storage.mount function or with the
See apyfal.storage for information on possible parameters for the targeted storage.
The following examples show the registration of the
my_storage storage type.
This storage needs the following parameters to be mounted:
With mount function¶
The registration of
my_storage storage is performed as follows.
import apyfal.storage # Mount "my_provider.my_bucket" storage apyfal.storage.mount(storage_type='my_storage', client_id='my_client_id', secret_id='my_secret_id')
With configuration file¶
The registration of
my_storage storage is performed by adding a
subsection to the configuration file containing storage parameters.
[storage.my_storage] client_id = my_client_id secret_id = my_secret_id
See Configuration for more information on the configuration file.