RClone : Mount Google Drive on any File System

Background

Would it be nice to directly manipulate Google Drive as part of the OS File System ? Would it be nice to automate backups directly in the cloud ? Would it be nice to stop buying expensive external storage ? The idea is not to send a "all to cloud" message. But sometimes, this very handy to have quickly have access to some centralized storage. Well, I think so. But How ? Cloud Storage is object storage (S3 like), so each files are actually identified by a unique URL and a set of credentials granting or not certain rights. There is a way of course !

RClone

Introducing RClone. RClone is an Open Source initiative which provide S3-like access to many Cloud Providers. Today, more than 40 cloud providers (including Dropbox, Amazon S3, OVH, Google Drive, ownCloud, on-prem S3, ...) are supported by RClone. The wonderful thing RClone is achieving is just what we want : mounting the Cloud Storage endpoint on your file system exactly like a standard block storage device.

Cherry on the cake, RClone works on Windows, macOS, linux and FreeBSD ! 

This article is focusing on Raspberry Pi with a specific purpose. You probably know that I'm active on the 3D Printing field. I'm using OctoPi to remotely access my 3D Printer and I recently had some issues with my Pi and lost my OctoPi backups (yes they were local). So I decided to send the backups elsewhere, why not in the cloud ?

Installing RClone

First, you need to install RClone on the Raspberry Pi. This is probably the easiest part ;)

$ wget https://downloads.rclone.org/v1.57.0/rclone-v1.57.0-linux-arm.deb

--2022-01-30 13:51:24--  https://downloads.rclone.org/v1.57.0/rclone-v1.57.0-linux-arm.deb

Resolving downloads.rclone.org (downloads.rclone.org)... 2a02:24e0:8:61f9::1, 5.153.250.7

Connecting to downloads.rclone.org (downloads.rclone.org)|2a02:24e0:8:61f9::1|:443... connected.

HTTP request sent, awaiting response... 200 OK

Length: 13842474 (13M) [application/vnd.debian.binary-package]

Saving to: 'rclone-v1.57.0-linux-arm.deb'

rclone-v1.57.0-linux-arm.deb        100%[===================================>]  13.20M  4.89MB/s    in 2.7s

pi@octopi:~ $ sudo apt install ./rclone-v1.57.0-linux-arm.deb

Reading package lists... Done

Building dependency tree

Reading state information... Done

Note, selecting 'rclone' instead of './rclone-v1.57.0-linux-arm.deb'

[...]

Setting up rclone (1.57.0) ...

Processing triggers for man-db (2.8.5-2) ...

pi@octopi:~ $ rclone --version

rclone v1.57.0

- os/version: raspbian 10.11

- os/kernel: 5.10.63-v7l+ (armv7l)

- os/type: linux

- os/arch: arm

- go/version: go1.17.2

- go/linking: static

- go/tags: none

Now that rclone is installed, it's time to start the configuration !

$ rclone config

e) Edit existing remote

n) New remote

d) Delete remote

r) Rename remote

c) Copy remote

s) Set configuration password

q) Quit config

e/n/d/r/c/s/q> n

name> GDrive

Option Storage.

Type of storage to configure.

Enter a string value. Press Enter for the default ("").

Choose a number from below, or type in your own value.

 1 / 1Fichier

   \ "fichier"

 2 / Alias for an existing remote

   \ "alias"

[...]

16 / Google Drive

   \ "drive"

Storage> 16

Option client_id.

Google Application Client Id

Setting your own is recommended.

See https://rclone.org/drive/#making-your-own-client-id for how to create your own.

If you leave this blank, it will use an internal key which is low performance.

Enter a string value. Press Enter for the default ("").

client_id>

Ok, here we need to stop, sit back and relax. There are 2 ways to configure our Google Drive connectivity. Either we go the easy way (without creating a client_id) or we go the hard way. Creating dedicated Google API and OAuth credentials. If you are familiar with the Google Cloud Console, this is the best way to have performance. If you do not want to enter into this, you can use the default option with the existing public RClone client_id. Down side is it will be less performant. It is up to you to decide and see what is best for you. I have tested the default access and it is more than enough for me. I'm just storing backup files once at night ... No worries. However, if you want to sync massive files multiple times a day, it is probably worth reading the below. You also need to understand that above a certain limit of API calls, RClone is charged by Google.

For the easy way, just to scroll down to Easy Setup, otherwise, continue reading.

Complex Setup

Ok, so you are the hardcore one, that's cool ! So, rclone is now asking for a client_id. We need to create one on the Google Cloud Console.

Here is a procedure from the rclone website (https://rclone.org/drive/#making-your-own-client-id). Trying to be as generic as possible because some sections are changing on the UI (Google redesigned some process and UI section).

Access the Google Cloud Console with the account you want to access the Google Drive

1. Select a project or create a new project.

2. Under "ENABLE APIS AND SERVICES" search for "Drive", and enable the "Google Drive API".

3. Click "Credentials" in the left-side panel (not "Create credentials", which opens the wizard), then "Create credentials"

4. If you already configured an "Oauth Consent Screen", then skip to the next step; if not, click on "CONFIGURE CONSENT SCREEN" button (near the top right corner of the right panel), then select "External" and click on "CREATE"; on the next screen, enter an "Application name" ("rclone" is OK) then click on "Save" (all other data is optional). Click again on "Credentials" on the left panel to go back to the "Credentials" screen.

(PS: if you are a GSuite user, you could also select "Internal" instead of "External" above, but this has not been tested/documented so far).

6. Click on the "+ CREATE CREDENTIALS" button at the top of the screen, then select "OAuth client ID".

7. Choose an application type of "Desktop app" and click "Create". (the default name is fine)

8. It will show you a client ID and client secret. Make a note of these.

9. Go to "Oauth consent screen" and press "Publish App"

10. Provide the noted client ID and client secret to rclone.

11. Click "OAuth consent screen", then click "PUBLISH APP" button and confirm, or add your account under "Test users".

Be aware that, due to the "enhanced security" recently introduced by Google, you are theoretically expected to "submit your app for verification" and then wait a few weeks(!) for their response; in practice, you can go right ahead and use the client ID and client secret with rclone, the only issue will be a very scary confirmation screen shown when you connect via your browser for rclone to be able to get its token-id (but as this only happens during the remote configuration, it's not such a big deal).

Enter a string value. Press Enter for the default ("").

client_id> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Option client_secret.

OAuth Client Secret.

Leave blank normally.

Enter a string value. Press Enter for the default ("").

client_secret> xxxxxxxxxxxxxxxxxxxxxxxxxxx

Now, you can jump to the common section below ;)

Easy Setup

So, at the question client_id, just push enter as well as client_secret and continue the below procedure.

Enter a string value. Press Enter for the default ("").

client_id> 

Option client_secret.

OAuth Client Secret.

Leave blank normally.

Enter a string value. Press Enter for the default ("").

client_secret> 

So again, this setup is sharing the rclone client_id with all the other existing setups and it will have an impact on performance since Google API calls are throttled. On top of that, there is an associated cost for the rclone developper. So, this is only recommended if you are testing the deployment. Once you are done, please create a client_id.

Common Setup

Option scope.

Scope that rclone should use when requesting access from drive.

Enter a string value. Press Enter for the default ("").

Choose a number from below, or type in your own value.

 1 / Full access all files, excluding Application Data Folder.

   \ "drive"

 2 / Read-only access to file metadata and file contents.

   \ "drive.readonly"

   / Access to files created by rclone only.

 3 | These are visible in the drive website.

   | File authorization is revoked when the user deauthorizes the app.

   \ "drive.file"

   / Allows read and write access to the Application Data folder.

 4 | This is not visible in the drive website.

   \ "drive.appfolder"

   / Allows read-only access to file metadata but

 5 | does not allow any access to read or download file content.

   \ "drive.metadata.readonly"

scope> 1

Option root_folder_id.

ID of the root folder.

Leave blank normally.

Fill in to access "Computers" folders (see docs), or for rclone to use

a non root folder as its starting point.

Enter a string value. Press Enter for the default ("").

root_folder_id>

Option service_account_file.

Service Account Credentials JSON file path.

Leave blank normally.

Needed only if you want use SA instead of interactive login.

Leading `~` will be expanded in the file name as will environment variables such as `${RCLONE_CONFIG_DIR}`.

Enter a string value. Press Enter for the default ("").

service_account_file>

Edit advanced config?

y) Yes

n) No (default)

y/n> n

Use auto config?

 * Say Y if not sure

 * Say N if you are working on a remote or headless machine

y) Yes (default)

n) No

y/n> n

Option config_verification_code.

Verification code

Go to this URL, authenticate then paste the code here.

https://accounts.google.com/o/oauth2/auth?access_type=offline&client_id=2020000000044.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&response_type=code&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive&state=LJQ5si-1AxxxxxxxxxxE2Q

Enter a string value. Press Enter for the default ("").

At this stage, we need to open a browser and copy/paste the URL in green above (of course, not this exact URL but the one on your terminal session). It will ask permission to access the Google Drive for the specified account.

If you have multiple accounts, choose the right one authorizing access to Drive : 

Next, you need to allow rclone application :

When done, you will have an authorization string that must be copied on the rclone config :

Let's paste the code on rclone and continue the setup : 

config_verification_code> 4/1AXxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Configure this as a Shared Drive (Team Drive)?

y) Yes

n) No (default)

y/n> n

--------------------

[GDrive]

type = drive

scope = drive

token = {"access_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"}

team_drive =

--------------------

y) Yes this is OK (default)

e) Edit this remote

d) Delete this remote

y/e/d> y

Current remotes:

Name                 Type

====                 ====

GDrive               drive

e) Edit existing remote

n) New remote

d) Delete remote

r) Rename remote

c) Copy remote

s) Set configuration password

q) Quit config

e/n/d/r/c/s/q> q

So, we are done with the configuration !

The configuration is saved on $HOME/.config/rclone/rclone.conf

Mounting the Drive on the filesystem

Next step is to mount the Drive on the file system. We have to use the following RClone command to do so.

$ rclone mount --daemon GDrive: /home/pi/gdrive

$ df -h

Filesystem      Size  Used Avail Use% Mounted on

/dev/root        15G  2.1G   12G  16% /

devtmpfs        1.7G     0  1.7G   0% /dev

tmpfs           1.9G     0  1.9G   0% /dev/shm

tmpfs           1.9G  8.6M  1.9G   1% /run

tmpfs           5.0M  4.0K  5.0M   1% /run/lock

tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup

/dev/mmcblk0p1  253M   49M  204M  20% /boot

tmpfs           379M     0  379M   0% /run/user/1000

GDrive:        1.0P     0  1.0P   0% /home/pi/gdrive

Depending on your subscription, you can have between 15 GB to 1PB storage. Above, you see 1PB since this is my work subscription.

Here is another example with my personal paid subscription of 200 GB :

$ df -h

Filesystem      Size  Used Avail Use% Mounted on

[...]

GDrive1:         200G  148G   53G  74% /home/pi/gdrive

Now, you can either create folder and files on the Google Drive UI or from your Linux CLI. It works well either way. There are some refresh factor to take into consideration between the UI and the CLI but I never had big issues with it.

You can imagine all files operations are now possible on this mount point. You can also be creative and transfer files between 2 Google Drive accounts !

$ df -h | grep GDrive

GDrive1:        200G  148G   53G  74% /home/me/GoogleDrivePerso

GDrive2:        1.0P     0  1.0P   0% /home/me/GoogleDriveWork

me@localhost:~/GoogleDrivePerso/ $ cp * ../GoogleDriveWork/

Of course it works ;)

Also, why not syncing your personal Google Drive to Dropbox ! You are only limited by your own imagination and use cases.

To detach the drive, there are no specific rclone command, you need to ask root to use the standard umount command : 

$ sudo umount GDrive:

Limitations

Of course, object storage is different from block storage. The purpose is not the same, and so are the capabilities. Objects are less performant - since they are used for archiving data and long term storage - than files stored as block. So, there are some limitations.

Supported operations on rclone drives are :

• Backup (and encrypt) files to cloud storage
• Restore (and decrypt) files from cloud storage
• Mirror cloud data to other cloud services or locally
• Migrate data to cloud, or between cloud storage vendors
• Mount multiple, encrypted, cached or diverse cloud storage as a disk
• Analyse and account for data held on cloud storage using lsf, ljson, size, ncdu
• Union file systems together to present multiple local and/or cloud file systems as one

Operation not supported : open a file for exclusive writing from a specific process. Example : OctoPrint cannot directly write on a RClone folder.

So, how am I sorting my backup concern ? I'm leaving backup at the original place and then I have a cron copying them on my Google Drive folder. That works fine so far !

$ crontab -l

# For more information see the manual pages of crontab(5) and cron(8)

#

# m h  dom mon dow   command

0 1 * * * cp /home/pi/.octoprint/data/backup/* /home/pi/GoogleDrivePerso/OctoPiBackups/

So, every night at 1 am, my latest OctoPrint backup is transfered to my Google Drive account. How cool is this ?

I hope this help. It is always nice to have some easy to use cloud storage.