Connecting to External Data Services

Algorithmia's Hosted Data storage is useful, but sometimes you already have your files stored with another provider such as Dropbox or Amazon S3, and you don't want to manually transfer them. Fortunately, you can use our Data Connectors to make a direct connection between your Algorithmia account and your files stored elsewhere.

When you see a Data URL which begins with a protocol other than "data://", such as "s3://" or "dropbox://", it indicates that this file is being accessed via a Data Connector. Nothing else changes: you can still send these files as input to Algorithms just as you would any other Data URL.

To begin, head to your Data Sources page and click the "New Data Source" menu in the upper-right. Pick the provider you're trying to connect to, such as Amazon S3, Google Cloud Storage, Azure Blob Storage, or Dropbox.

Next, enter your account details and set up your connection. The specifics of how to connect to each provider can be found in our Data Portal Guides, but they all contain the following options:

Label: a name you choose to remind you of what this connects to.

Path Restriction: enter * to allow any file to be accessed, or enter a path such as /images/avatars/* if you only want to access files under the "/images/avatars/" subfolder.

Allow writing: by default, connectors are read-only; check this box to allow files to be written to (and deleted from) this connected data source.

Make default: if checked, this becomes the default connector for that protocol (e.g. any "dropbox://" path goes to this connector when used from your account). If unchecked, the Label is added to the protocol ("dropbox+randomname://") to distinguish it from other connectors of the same type, since you can create as many different Connectors as you wish.

To delete a connector, simply click on it from your Data Sources page, then click the "Disconnect Source" button. This doesn't affect your files on the provider in any way -- it just removes your ability to connect to them from Algorithmia.

Note: some Algorithms, such as DeepFilter, allow you to specify an output location for the results ("savePaths", in this case). This can be especially useful when using a Data Connector. For example, I could connect my Dropbox account, drag several images into a Dropbox folder, then use that folder as both the input and the output location for my DeepFilter images. When I run the Algor ithm, the results show up right there in my Dropbox!

If you prefer to move files around via code, instead of using a web interface, you'll be excited to learn about our Data API.

 


NEXT