Programmatic Access


There are several ways to access the metadata and data on the 4DN data portal. The most straightforward and intuitive ways to do so are via the search and browse functionality on the web site.

search box search box

browse menu browse menu

The web site also facilitates downloading files individually or in bulk from data sets of interest.

However, as you become more familiar with the data sets available from the portal you may want to programmatically access metadata and data. Additionally, while we provide workbooks and easy to use tools for data submission some advanced users may wish to do submissions programmatically.

The sections below explain some of the options available.

DCIC utils

If you program in Python the 4DN-DCIC has developed and maintains a package of utilities that contains useful functions for accessing 4DN metadata from the portal. You can install it with pip install dcicutils

To get started using this package and for some information on useful functions to access metadata see the documentation here

If you've already installed the package and want to see some examples of how you can use it look here

If you don't use Python expand the REST API section below.


Downloading Files

File download from the 4DN data portal now requires authentication, even if the file is public. Accounts can be created by anyone, including those not part of the 4DN Network; for more information on account creation, see the Account Creation page. Files can be downloaded from the web portal after logging in, as usual; for file download from the command line, the process is described below.

First you need to create a new access key, if you don't have an access key already. Note that access keys created for Jupyterhub can't be used for the rest of the portal.

To create a new access key, first log in to the data portal, then click on your account in the upper right and click on Profile from the dropdown menu. There will be a button near the bottom of the page to add an access key. Save the key and secret; typically this is done by creating a file in your home directory called keypairs.json with the following contents/format (replacing the x’s with the appropriate key and secret, of course):

    "default": {
        "key": "XXXXXXXX",
        "secret": "xxxxxxxxxxxxxxxx",
        "server": ""

Once the access key is created and stored, the file can be downloaded via curl with the following command:

curl -O -L --user <key>:<secret> <download-url>

For more information on file downloads see this page