Skip to content

File numbers and file sizes in Sunet Drive

A common question is how many files can I handle and how large can the files be? And the answer generally is: “It depends…”

Sunet Drive is based on Nextcloud and can handle about as much and as many as other Enterprise File Sync And Share (EFSS) solutions like Dropbox or Google Drive. The general recommendation is that if you have more than 100,000 files or files larger than 4 GB, you should consult with your local institution and define a proper data management plan, since these parameters can affect the complexity of handling the data in the long run. The same applies to data that requires frequent changes or especially fast access to the data. Here are a few rules that can be applied when handling larger datasets:

  • Projects with more than 100,000 files should be split into separate parts consisting of less than 100,000 files per part.
  • The absolute file size limit in Sunet Drive is 30 GB
  • S3-buckets can handle millions of files, but clients should only selectively synchronize the files currently needed.
  • Try to avoid creating more than 10,000 folders, especially empty folders affect performance negatively
  • Do not store more than 1TB total in one S3 bucket, if you have more data than that in one project, request more buckets for that project.
  • If you need fast access to your files, synchronize them locally to a fast hard disk (SSD).
  • Expect to invest time when handling large files, especially files larger than 4GB are known to cause problems with some client applications. Sunet Drive is not suitable e.g, hard drives images or images of virtual machines.
  • Databases (i.e., files associated with a live database instance) should not be added to Sunet Drive. There are more suitable solutions to create backups of databases. Uploading database dumps is fine, as long as they are within the other limits.
  • If you work with source code and/or software development, you should use a proper version control system like git or subversion.

If you are in doubt, you can always test your scenario with non-production data and consult with your local IT.