Geospatial Solutions Expert: Working with Dropbox API in python

In this post, I will explore Dropbox API. If you don't know what Dropbox is, according to its Wikipedia page: "Dropbox is a file hosting service operated by the American company Dropbox, Inc., headquartered in San Francisco, California, U.S. that offers cloud storage, file synchronization, personal cloud, and client software".

Now that you know what dropbox is, lets see how we can do some basic operations such as uploading, renaming, create, copy, move, delete, download, list to files and folders using python API.

First create an app to generate an API TOKEN on the app developer console page after setting up the right permissions for you use case.

There are two common ways/methods of connecting to dropbox via python:-

The first is using a third party module named "dropbox" which you can install using pip install dropbox. The documentation is found on Dropbox for Python web page.

The second way is using requests module based on their official Dropbox API Explorer.

Which ever method you decide to adopt, it is just a matter of preference. In my case I usually use the combination of the two depending on what is easier for what I want to implement.

Both have excellent documentations. In fact, the Dropbox API Explorer is fantastic for visually constructing the end calls you wanted. For example, I want to get list of contents in a folder, I will on lis_folder tab on the left and configure the options as seen. Then click on show code to grab the code for use in python environment.

Just copy the resulting code into your python environment to execute the API.

import requests
import json

url = "https://api.dropboxapi.com/2/files/list_folder"

headers = {
    "Authorization": "Bearer <access-token>",
    "Content-Type": "application/json"
}

data = {
    "path": "/Images"
}

r = requests.post(url, headers=headers, data=json.dumps(data))

Lets take a look at more specific examples:-

The dropbox folder am going to use in this article is this root folder at: "/Cat Watching Test". It contains subfolders and files as seen below;

Lets explore the folder and its contents using API... For each exploration, I will use both requests module (Dropbox API Explorer) and dropbox module (Dropbox for Python). Lets get our hands dirty.

1. List all files and folders in a directory

Here we want to get the names of all subfolders and files in our root folder. Using the dropbox api explorer, the configuration will look like this;

Note that I set the "recursive" option to "true", this will recursively get access to contents of nested subfolders within the main/root folder.

The result python code is like this:-

# Using requests module (Dropbox API Explorer)

import requests
import json

url = "https://api.dropboxapi.com/2/files/list_folder"

headers = {
    "Authorization": "Bearer <access-token>",
    "Content-Type": "application/json"
}

data = {
    "path": "/Cat Watching Test",
    "recursive": True
}

r = requests.post(url, headers=headers, data=json.dumps(data))

This code will return a JSON object with the following keys "entries", "cursor" and "has_more". "entries" is where you will find all files and folders together with their properties a seen below.

While the "cursor" and "has_more" keys are used to access more content on next page if your folder has huge content, "has_more" will has a values of 'true' and you can use the "cursor" in list_folder/continue endpoint to get contents from the next page.

The JSON: you can manipulate it using normal python function to extract the needed datasets.

{
  "entries": [
    {
      ".tag": "folder",
      "name": "Cat Watching Test",
      "path_lower": "/cat watching test",
      "path_display": "/Cat Watching Test",
      "id": "id:DkfHLtGtYtAAAAAAAAACrg"
    },
    {
      ".tag": "folder",
      "name": "Cat1",
      "path_lower": "/cat watching test/cat1",
      "path_display": "/Cat Watching Test/Cat1",
      "id": "id:DkfHLtGtYtAAAAAAAAAGcA"
    },
    {
      ".tag": "folder",
      "name": "Cat2",
      "path_lower": "/cat watching test/cat2",
      "path_display": "/Cat Watching Test/Cat2",
      "id": "id:DkfHLtGtYtAAAAAAAAAGdQ"
    },
    {
      ".tag": "folder",
      "name": "Cat3",
      "path_lower": "/cat watching test/cat3",
      "path_display": "/Cat Watching Test/Cat3",
      "id": "id:DkfHLtGtYtAAAAAAAAAGeg"
    },
    {
      ".tag": "file",
      "name": "cat1.txt",
      "path_lower": "/cat watching test/cat1.txt",
      "path_display": "/Cat Watching Test/cat1.txt",
      "id": "id:DkfHLtGtYtAAAAAAAAACrQ",
      "client_modified": "2022-09-28T22:26:20Z",
      "server_modified": "2022-09-28T22:26:21Z",
      "rev": "5e9c4401000ae38a297eb",
      "size": 24,
      "is_downloadable": true,
      "content_hash": "8200f0ef3c83c0dcdc8140545c49d063c12ccabfffa4b8c44f4845828ce893d5"
    },
    {
      ".tag": "file",
      "name": "cat.jpg",
      "path_lower": "/cat watching test/cat1/cat.jpg",
      "path_display": "/Cat Watching Test/Cat1/cat.jpg",
      "id": "id:DkfHLtGtYtAAAAAAAAAGcw",
      "client_modified": "2022-10-06T14:43:15Z",
      "server_modified": "2022-10-06T14:43:15Z",
      "rev": "5ea5eb6a042db38a297eb",
      "size": 69510,
      "is_downloadable": true,
      "content_hash": "53297d0b1f69b5ef0fe801f72fa420555d3338b796e8ac1f67f3d591f0dfdcad"
    },
    {
      ".tag": "file",
      "name": "new_cat123.jpg",
      "path_lower": "/cat watching test/cat1/new_cat123.jpg",
      "path_display": "/Cat Watching Test/Cat1/new_cat123.jpg",
      "id": "id:DkfHLtGtYtAAAAAAAAAGcg",
      "client_modified": "2022-10-06T14:43:13Z",
      "server_modified": "2022-10-06T14:57:47Z",
      "rev": "5ea5eea9bb94c38a297eb",
      "size": 69085,
      "is_downloadable": true,
      "content_hash": "41b77df3862f0486810180e7f8a31eba9e83c7a4b1aebedde17dc7a337227b98"
    },
    {
      ".tag": "file",
      "name": "NEW_CAT_6.jpg",
      "path_lower": "/cat watching test/cat1/new_cat_6.jpg",
      "path_display": "/Cat Watching Test/Cat1/NEW_CAT_6.jpg",
      "id": "id:DkfHLtGtYtAAAAAAAAAGcQ",
      "client_modified": "2022-10-06T14:43:10Z",
      "server_modified": "2022-10-06T16:09:29Z",
      "rev": "5ea5feb00974c38a297eb",
      "size": 40066,
      "is_downloadable": true,
      "content_hash": "f795b542189b8ebb8a8e4f4eb4d41635f277b6a8e74a930ad3fae896f16adb26"
    },
    {
      ".tag": "file",
      "name": "NEW_CAT_7.jpg",
      "path_lower": "/cat watching test/cat1/new_cat_7.jpg",
      "path_display": "/Cat Watching Test/Cat1/NEW_CAT_7.jpg",
      "id": "id:DkfHLtGtYtAAAAAAAAAGdA",
      "client_modified": "2022-10-06T14:43:15Z",
      "server_modified": "2022-10-06T16:09:32Z",
      "rev": "5ea5feb2eb99f38a297eb",
      "size": 90444,
      "is_downloadable": true,
      "content_hash": "4037dfcc32d116a3ea49e322c9c7bf83e39162c1e4ff247019bdf94bff5329ff"
    },
    {
      ".tag": "file",
      "name": "NEW_CAT_10.jpg",
      "path_lower": "/cat watching test/cat2/new_cat_10.jpg",
      "path_display": "/Cat Watching Test/Cat2/NEW_CAT_10.jpg",
      "id": "id:DkfHLtGtYtAAAAAAAAAGdg",
      "client_modified": "2022-10-06T14:43:37Z",
      "server_modified": "2022-10-06T16:09:34Z",
      "rev": "5ea5feb59933d38a297eb",
      "size": 31623,
      "is_downloadable": true,
      "content_hash": "acdc93a1d2e385af21e4511bf7890e936827afc6f2b22eea006cdcdc177a773a"
    },
    {
      ".tag": "file",
      "name": "NEW_CAT_11.jpg",
      "path_lower": "/cat watching test/cat2/new_cat_11.jpg",
      "path_display": "/Cat Watching Test/Cat2/NEW_CAT_11.jpg",
      "id": "id:DkfHLtGtYtAAAAAAAAAGdw",
      "client_modified": "2022-10-06T14:43:39Z",
      "server_modified": "2022-10-06T16:09:37Z",
      "rev": "5ea5feb7f56f238a297eb",
      "size": 31618,
      "is_downloadable": true,
      "content_hash": "48e527965139a8334959f694d0da26f32ffd1890177f7c39ff1d598a2521dfae"
    },
    {
      ".tag": "file",
      "name": "NEW_CAT_8.jpg",
      "path_lower": "/cat watching test/cat2/new_cat_8.jpg",
      "path_display": "/Cat Watching Test/Cat2/NEW_CAT_8.jpg",
      "id": "id:DkfHLtGtYtAAAAAAAAAGeA",
      "client_modified": "2022-10-06T14:43:40Z",
      "server_modified": "2022-10-06T16:09:39Z",
      "rev": "5ea5feb9e4fa638a297eb",
      "size": 60566,
      "is_downloadable": true,
      "content_hash": "39fe352dfc3e4247454d6ab89d3be3ac866263c002f6d9b5ee923c9268a73e0d"
    },
    {
      ".tag": "file",
      "name": "NEW_CAT_9.jpg",
      "path_lower": "/cat watching test/cat2/new_cat_9.jpg",
      "path_display": "/Cat Watching Test/Cat2/NEW_CAT_9.jpg",
      "id": "id:DkfHLtGtYtAAAAAAAAAGeQ",
      "client_modified": "2022-10-06T14:43:41Z",
      "server_modified": "2022-10-06T16:09:41Z",
      "rev": "5ea5febbb899338a297eb",
      "size": 359035,
      "is_downloadable": true,
      "content_hash": "5507f7df934f0f72f5ced2dc3ad3417699ee93b48832db2751ec04620b07e36b"
    },
    {
      ".tag": "file",
      "name": "NEW_CAT_1.jpg",
      "path_lower": "/cat watching test/cat3/new_cat_1.jpg",
      "path_display": "/Cat Watching Test/Cat3/NEW_CAT_1.jpg",
      "id": "id:DkfHLtGtYtAAAAAAAAAGew",
      "client_modified": "2022-10-06T14:43:50Z",
      "server_modified": "2022-10-06T16:09:43Z",
      "rev": "5ea5febdd5d6f38a297eb",
      "size": 32128,
      "is_downloadable": true,
      "content_hash": "c1b4b68199d9ac21e961b388b0888d9b4c417634e0eb9a0e9c9eb0d22aa50dc0"
    },
    {
      ".tag": "file",
      "name": "NEW_CAT_3.jpg",
      "path_lower": "/cat watching test/cat3/new_cat_3.jpg",
      "path_display": "/Cat Watching Test/Cat3/NEW_CAT_3.jpg",
      "id": "id:DkfHLtGtYtAAAAAAAAAGfA",
      "client_modified": "2022-10-06T14:43:54Z",
      "server_modified": "2022-10-06T16:09:45Z",
      "rev": "5ea5fec01138b38a297eb",
      "size": 283169,
      "is_downloadable": true,
      "content_hash": "42ff363ce8ec6946ce7b5e95d6d85d62d096255658d10097f3bed1806dd5e75e"
    },
    {
      ".tag": "file",
      "name": "NEW_CAT_4.gif",
      "path_lower": "/cat watching test/cat3/new_cat_4.gif",
      "path_display": "/Cat Watching Test/Cat3/NEW_CAT_4.gif",
      "id": "id:DkfHLtGtYtAAAAAAAAAGfQ",
      "client_modified": "2022-10-06T14:43:55Z",
      "server_modified": "2022-10-06T16:09:48Z",
      "rev": "5ea5fec249cd138a297eb",
      "size": 486579,
      "is_downloadable": true,
      "content_hash": "bd68d305d7ccc5be4650f59b9d936bb4020b1f1c337df34b30e6d74bd4cbdc82"
    },
    {
      ".tag": "file",
      "name": "NEW_CAT_2.jpg",
      "path_lower": "/cat watching test/cat3/new_cat_2.jpg",
      "path_display": "/Cat Watching Test/Cat3/NEW_CAT_2.jpg",
      "id": "id:DkfHLtGtYtAAAAAAAAAGfg",
      "client_modified": "2022-10-06T14:44:00Z",
      "server_modified": "2022-10-06T16:09:50Z",
      "rev": "5ea5fec476c2d38a297eb",
      "size": 1275905,
      "is_downloadable": true,
      "content_hash": "be1604b3ed962b1100dc3dde4bb320dc776a097e8357e467ce07648c5113e342"
    }
  ],
  "cursor": "AAEAqAH-son4PzpkXrCh7tZpDeqUNTM71d7nxWIUss3OYc7HyroBI-VFcG6ujHSi_jwJH8FdnRPry2N6WRrNPYcvM6W9RR_eE3RqTDaVUEZY8MdDD0vl0ELbnOWClZmLA26TRhr6ZTMd-zLaaeKvWhiYbvAmOEhsyUiwr1UHfNvVZ2NKVBoF1L-MzGG0sfwCUR34at6_Qwap1Ma-nHzIcu_GMyhZBuT_KEhq7iZWcT7zPA",
  "has_more": false
}

Using dropbox for python module:-

# Using dropbox module (Dropbox for Python)
import dropbox
from dropbox import Dropbox

DROPBOX_ACCESS_TOKEN = 'Your Access Token'

# Create a connection dbx via API...
my_client = Dropbox(DROPBOX_ACCESS_TOKEN)
print("[SUCCESS] dropbox account linked")

# Read all subfolders in a directory...
file_list = my_client.files_list_folder(r'/Cat Watching Test', recursive=True)

# List of folders/files... by checking if type is a dbx folder/file...
myfolders = [x.path_display for x in file_list.entries if type(x) is dropbox.files.FolderMetadata]
myfiles = [x.name for x in file_list.entries if type(x) is dropbox.files.FileMetadata]

len(myfolders), len(myfiles)

2. Move file/folder from on path to another

Note that if the new path is the same as the old path, the file/folder get renamed. Lets see how we can move the contents of "/Cat Watching Test/Cat1" and "/Cat Watching Test/Cat2".

# Using requests module (Dropbox API Explorer)

import requests
import json

url = "https://api.dropboxapi.com/2/files/move_v2"

headers = {
    "Authorization": "Bearer <access-token>",
    "Content-Type": "application/json"
}

data = {
    "from_path": "/Cat Watching Test/Cat1/cat.jpg",
    "to_path": "/Cat Watching Test/Cat2/cat.jpg"
}

r = requests.post(url, headers=headers, data=json.dumps(data))

# ---------------------------------------------------


# Returned JSON object...

{
  "metadata": {
    ".tag": "file",
    "name": "cat.jpg",
    "path_lower": "/cat watching test/cat2/cat.jpg",
    "path_display": "/Cat Watching Test/Cat2/cat.jpg",
    "id": "id:DkfHLtGtYtAAAAAAAAAGcw",
    "client_modified": "2022-10-06T14:43:15Z",
    "server_modified": "2022-10-08T06:55:18Z",
    "rev": "5ea8068d0600738a297eb",
    "size": 69510,
    "is_downloadable": true,
    "content_hash": "53297d0b1f69b5ef0fe801f72fa420555d3338b796e8ac1f67f3d591f0dfdcad"
  }
}

# Using dropbox module (Dropbox for Python)
import dropbox
from dropbox import Dropbox

dropbox_path = r'/Cat Watching Test/Cat1/cat.jpg'
new_file_path = r'/Cat Watching Test/Cat2/cat.jpg' # dropbox_path.replace('cat_5', 'new_cat123')


# Create a connection dbx via API...
client = dropbox.Dropbox(DROPBOX_ACCESS_TOKEN)
print("[SUCCESS] dropbox account linked")


# Rename the file
client.files_move_v2(dropbox_path, new_file_path)

print("[MOVING/RENAMING] done with success...")

Note: the code above can also be used for renaming files, if the new and old file path names are the same then the former gets overwritten.

3. Uploading files from local disc to dropbox

Here we will transfer a file on local computer to the cloud on dropbox. You have to provide the complete file paths on both dropbox and local disc.

In this case dropbox path is: /Cat Watching Test/Cat1/Whatappimage.jpeg

import requests
import json

url = "https://content.dropboxapi.com/2/files/upload"

headers = {
    "Authorization": "Bearer <access-token>",
    "Content-Type": "application/octet-stream",
    "Dropbox-API-Arg": "{\"path\":\"/Cat Watching Test/Cat1/Whatappimage.jpeg\"}"
}

data = open("WhatsApp Image 2022-10-03 at 9.03.02 PM.jpeg", "rb").read()

r = requests.post(url, headers=headers, data=data)

# -----------------------------------------------------------------------


# Resulting JSON object...

{
  "name": "Whatappimage.jpeg",
  "path_lower": "/cat watching test/cat1/whatappimage.jpeg",
  "path_display": "/Cat Watching Test/Cat1/Whatappimage.jpeg",
  "id": "id:DkfHLtGtYtAAAAAAAAAGgA",
  "client_modified": "2022-10-08T07:21:45Z",
  "server_modified": "2022-10-08T07:21:46Z",
  "rev": "5ea80c76dec6038a297eb",
  "size": 49416,
  "is_downloadable": true,
  "content_hash": "fce6d01f98ff75f88814e76520b3f7345091e6332c9c75b2300354a95876a9d6"
}

# Using dropbox module (Dropbox for Python)
import dropbox
from dropbox import Dropbox

dropbox_path = r"/Cat Watching Test/Cat1/cat_NEWUPLOAD.jpg"
computer_path = r"cat_image\Cat1\cat_NEWUPLOAD.jpg"

# Create a connection dbx via API...
client = dropbox.Dropbox(DROPBOX_ACCESS_TOKEN)
print("[SUCCESS] dropbox account linked")

# Use files_upload() method...
client.files_upload(open(computer_path, "rb").read(), dropbox_path)
print("[UPLOADED] {}".format(computer_path))

4. Delete file/folder

# Using requests module (Dropbox API Explorer)

import requests
import json

url = "https://api.dropboxapi.com/2/files/delete_v2"

headers = {
    "Authorization": "Bearer <access-token>",
    "Content-Type": "application/json"
}

data = {
    "path": "/Cat Watching Test/Cat1/Whatappimage.jpeg"
}

r = requests.post(url, headers=headers, data=json.dumps(data))

# ===========================================================


# Returned JSON....

{
  "metadata": {
    ".tag": "file",
    "name": "Whatappimage.jpeg",
    "path_lower": "/cat watching test/cat1/whatappimage.jpeg",
    "path_display": "/Cat Watching Test/Cat1/Whatappimage.jpeg",
    "id": "id:DkfHLtGtYtAAAAAAAAAGgA",
    "client_modified": "2022-10-08T07:21:45Z",
    "server_modified": "2022-10-08T07:21:46Z",
    "rev": "5ea80c76dec6038a297eb",
    "size": 49416,
    "is_downloadable": true,
    "content_hash": "fce6d01f98ff75f88814e76520b3f7345091e6332c9c75b2300354a95876a9d6"
  }
}

# Using dropbox module (Dropbox for Python)
import dropbox
from dropbox import Dropbox


dropbox_path = r"/Cat Watching Test/Cat1/Umar"

# Create a connection dbx via API...
client = dropbox.Dropbox(DROPBOX_ACCESS_TOKEN)
print("[SUCCESS] dropbox account linked")

# Delete file...
client.files_delete(dropbox_path, parent_rev=None)
print("Files DELETED...")

5. Creating a folder

You can create nested folder structure, just provide the path as you wanted where each folder name is followed by forward slash (/).

Using requests module (Dropbox API Explorer):

import requests
import json

url = "https://api.dropboxapi.com/2/files/create_folder_v2"

headers = {
    "Authorization": "Bearer <access-token>",
    "Content-Type": "application/json"
}

data = {
    "path": "/Cat Watching Test/Cat1/test123"
}

r = requests.post(url, headers=headers, data=json.dumps(data))

# --------------------------------------------------------------


# Resulting JSON object...

{
  "metadata": {
    "name": "test123",
    "path_lower": "/cat watching test/cat1/test123",
    "path_display": "/Cat Watching Test/Cat1/test123",
    "id": "id:DkfHLtGtYtAAAAAAAAAGgQ"
  }
}

Using dropbox module (Dropbox for Python):

# Using dropbox module (Dropbox for Python)
import dropbox
from dropbox import Dropbox


dropbox_path = r"/Cat Watching Test/Cat1/Umar/Yusuf/Hassan/Tim"

# Create a connection dbx via API...
client = dropbox.Dropbox(DROPBOX_ACCESS_TOKEN)
print("[SUCCESS] dropbox account linked")

# Use files_upload() method...
client.files_create_folder_v2(dropbox_path, autorename=False)
print("Folder CREATED...")

6. Recursively Get Files and Folders Metadata as JSON

T = True
i = 1
dbx_token = 'el.W450xCfahkYilw6529nmnk81UI0'


url = "https://api.dropboxapi.com/2/files/list_folder"

headers = {
    "Authorization": f"Bearer {dbx_token}",
    "Content-Type": "application/json"
}

data = {
    "path": "/Root Folder on Dropbox/other folders",
    "recursive": True
}

r = requests.post(url, timeout=10, headers=headers, data=json.dumps(data))
reslt = r.json()
cur = reslt['cursor']

# Write to file....
with open(fr'JSON folder\\IB\\0.json', 'w') as outfile:
    json.dump(reslt['entries'], outfile)

# ------------------------------------------------------
# ------------------------------------------------------


while T:
    print('Processing....', i)
    url = "https://api.dropboxapi.com/2/files/list_folder/continue"

    headers = {
        "Authorization": f"Bearer {dbx_token}",
        "Content-Type": "application/json"
    }

    data = {
        "cursor": cur
    }
    
    r2 = requests.post(url, timeout=10, headers=headers, data=json.dumps(data))
    
    reslt2 = r2.json()
    cur = reslt2['cursor']
    
    # Write to file....
    with open(fr'JSON folder\\IB\\{i}.json', 'w') as outfile:
        json.dump(reslt2['entries'], outfile)
        
    if reslt2['has_more'] == False:
        T = False
        
    i = i+1
    
print('Done...')

The code below will then go into the JSON folder and combine the dataset into a single table.

jdf_list = []

for i in range(1, 283):
    print(i)
    
    # Opening JSON file
    with open(f'JSON folder/IB/{i}.json', 'r') as openfile:
        # Reading from json file
        json_object = json.load(openfile)
    
    # Make df and add to list of df...
    jdf = pd.DataFrame(json_object)
    jdf_list.append(jdf)

    
# Combine the metadata into a single table...
jdf_list_merge = pd.concat(jdf_list)

# jdf_list_merge.to_excel('filename.xlsx', index=False)
jdf_list_merge.shape

This is not in anyway an exhaustive list of what you can do with the dropbox API. There are many operations you can however, above are some of the common once and understanding the concept behind them will open your eyes to working with others that are covered above.

Thank you for reading.

Geospatial Solutions Expert

Wednesday, October 19, 2022

Working with Dropbox API in python