How to clean archive folder properly

johnlabor

Hi,

I used to save a lot of photos around 2GB on cryptpad.
Later on, I migrate them to another data storage, so I go ahead and use destroy function to delete all those images.
They did get removed from the web app and the storage bar showed I only have 500mb of data now.

However, when I check the underlying filesystem on the server, data/archive still have those 2GB worth of objects on disk. I cant seem to find any documentation related to the ways of cleaning this folder.

Can I ask if someone know what is the best way to clean archive folder? I want to make sure if I delete the contents of that folder will not cause any actual data loss.

Thanks!

Mathilde

johnlabor Can I ask if someone know what is the best way to clean archive folder? I want to make sure if I delete the contents of that folder will not cause any actual data loss.

Hello,

It depends on your instance retention policies. We will enrich the current documentation with more information for the next release, work as already been prepared on this commit in our GitHub repository.

Hope this helps!

johnlabor

Hi @Mathilde is there any pointer from you on this question?
Appreciated! Thanks.

johnlabor

Hi @Mathilde thanks for the reply here.

However, these scripts does not clean up anything in my archive folder and shows as 0 blob to clean.
If I directly delete folders within ./data/archive/* will that cause any issues?

Thanks.

johnlabor

After diving into the code I found out in lib/storage/blob.js the archive path is not found at all:

[Error: ENOENT: no such file or directory, scandir 'data/archive/blob'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'scandir',
  path: 'data/archive/blob'
}

I hardcode this return in prependArchive func with:

    return Path.join('/cryptpad', Env.archivePath, 'blob', relativePathToBlob);

And finally runs:

["INFO","2024-05-02T04:24:43.665Z","EVICT_ARCHIVED_FINAL_REPORT",{"launchTime":1714623881510,"archivedChannelsRemoved":0,"archivedAccountsRemoved":0,"archivedBlobProofsRemoved":326,"archivedBlobsRemoved":326}]

Really painful process.
At least it removes 2.7G worth of old data now.

Thanks.

Mathilde

johnlabor interesting. According to our main engineer, you may have ran the script outside the cryptpad directory, hence it couldn't access the data/ folder. Did you run it this way?

node cryptpad/scripts/evict-archived.js

The script should be run from within the cryptpad/ folder, not outside. You may want to revert the changes you made to the source code in lib/storage/blob.js. It will cause you issues next time you want to upgrade to a newer version!

In any case, glad you got it working! 🙂

ZoopaMario

Hello 🙂
I seem to have a similar problem trying to clean up the archive folder in my Docker setup. Its been ~ 5GB for a while now and i assumed it should be cleaned up automatically over time. Manually executing the evict scripts shows me it does look through the data though does not delete anything:

$ docker exec cryptpad /usr/local/bin/node scripts/evict-inactive.js --workdir=/cryptpad

["INFO","2024-07-14T09:50:15.254Z","EVICT_CHANNEL_ACTIVITY_START","Assessing channel activity"]
["INFO","2024-07-14T09:50:15.555Z","EVICT_CHANNEL_CATEGORIZATION_PROGRESS",{"channels":1000}]
["INFO","2024-07-14T09:50:15.635Z","EVICT_CHANNELS_CATEGORIZED",{"active":1477,"channels":1477}]
["INFO","2024-07-14T09:50:15.641Z","EVICT_BLOBS_ACTIVITY_START","Assessing blob activity"]
["INFO","2024-07-14T09:50:16.292Z","EVICT_BLOB_CATEGORIZATION_PROGRESS",{"blobs":1000}]
["INFO","2024-07-14T09:50:16.844Z","EVICT_BLOB_CATEGORIZATION_PROGRESS",{"blobs":2000}]
["INFO","2024-07-14T09:50:17.398Z","EVICT_BLOB_CATEGORIZATION_PROGRESS",{"blobs":3000}]
["INFO","2024-07-14T09:50:17.623Z","EVICT_BLOBS_CATEGORIZED",{"active":3387,"blobs":3387}]
["INFO","2024-07-14T09:50:17.628Z","EVICT_ACCOUNTS_ACTIVITY_START","Assessing account activity"]
["INFO","2024-07-14T09:50:24.567Z","EVICT_COUNT_ACCOUNTS",{"accounts":45,"inactive":0}]
["INFO","2024-07-14T09:50:24.568Z","EVICT_BLOB_START",{}]
["INFO","2024-07-14T09:50:25.182Z","EVICT_BLOB_PROGRESS",{"blobs":1000}]
["INFO","2024-07-14T09:50:25.737Z","EVICT_BLOB_PROGRESS",{"blobs":2000}]
["INFO","2024-07-14T09:50:26.281Z","EVICT_BLOB_PROGRESS",{"blobs":3000}]
["INFO","2024-07-14T09:50:26.494Z","EVICT_BLOBS_REMOVED",0]
["INFO","2024-07-14T09:50:26.499Z","EVICT_ARCHIVE_INACTIVE_BLOB_PROOFS_START",{}]
["INFO","2024-07-14T09:50:27.118Z","EVICT_BLOB_PROOF_PROGRESS",{"proofs":1000}]
["INFO","2024-07-14T09:50:27.644Z","EVICT_BLOB_PROOF_PROGRESS",{"proofs":2000}]
["INFO","2024-07-14T09:50:28.194Z","EVICT_BLOB_PROOF_PROGRESS",{"proofs":3000}]
["INFO","2024-07-14T09:50:28.418Z","EVICT_BLOB_PROOFS_REMOVED",{"removed":0,"total":3387}]
["INFO","2024-07-14T09:50:28.425Z","EVICT_INACTIVE_CHANNELS_START",{}]
["INFO","2024-07-14T09:50:30.284Z","EVICT_INACTIVE_CHANNELS_PROGRESS",{"channels":1000,"archived":0}]
["INFO","2024-07-14T09:50:31.130Z","EVICT_CHANNELS_ARCHIVED",{"channels":1477,"archived":0}]
["INFO","2024-07-14T09:50:31.131Z","EVICT_TIME_TO_RUN_SCRIPT",15878]
["INFO","2024-07-14T09:50:31.133Z","EVICT_INACTIVE_FINAL_REPORT",{"launchTime":1720950615249,"activeChannels":1477,"totalChannels":1477,"totalBlobs":3387,"activeBlobs":3387,"totalAccounts":45,"activeAccounts":45,"channelsArchived":0,"runningTime":15878}]

$ docker exec -t cryptpad /usr/local/bin/node scripts/evict-archived.js --workdir=/cryptpad

["INFO","2024-07-14T09:51:43.973Z","EVICT_ARCHIVED_CHANNELS_REMOVED",0]
["INFO","2024-07-14T09:51:43.979Z","EVICT_ARCHIVED_ACCOUNTS_REMOVED",0]
["INFO","2024-07-14T09:51:45.107Z","EVICT_ARCHIVED_BLOB_PROOFS_REMOVED",0]
["INFO","2024-07-14T09:51:46.163Z","EVICT_ARCHIVED_BLOBS_REMOVED",0]
["INFO","2024-07-14T09:51:46.164Z","EVICT_ARCHIVED_FINAL_REPORT",{"launchTime":1720950703919,"archivedChannelsRemoved":0,"archivedAccountsRemoved":0,"archivedBlobProofsRemoved":0,"archivedBlobsRemoved":0}]

I've tried reducing the retention times and running the scripts again. Running evict-inactive.js added about 400MB to the archive folder, though it also threw multiple (["ERROR","2024-07-14T10:30:23.993Z","EVICT_BLOB_LIST_PROOFS_ERROR",{"code":"ENOENT","reason":"INACTIVE"}]) Errors and again nothing was actually deleted when running evict-archived.js (same output as before)

my config/config.js

...
     inactiveTime: 10,
     archiveRetentionTime: 5,
     accountRetentionTime: 180,
    //disableIntegratedEviction: true,
...

any help would be appreciated, alternatively it would be nice to know if its fine to just delete the contents of data/archive manually ?

EDIT: I just reduced the archiveRetentionTime to 0 and now running the evict-archived.js script actually did clean the archive folder. I have not deleted any large amount of files in a while so i 'm still a little confused why it didn't evict with a value of 5 (or automatically with the default value before i changed it). Anyway now that it works i can just automate it myself to make sure it runs

EDIT2: Also to answer @johnlabor and my own question, it should be fine to manually delete the archive folders contents, as it is also not listed as necessary for backup / migration in the docs