Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

problem with uploading a huge number of files #86

Open
inokaw opened this issue Jul 20, 2023 · 4 comments
Open

problem with uploading a huge number of files #86

inokaw opened this issue Jul 20, 2023 · 4 comments

Comments

@inokaw
Copy link

inokaw commented Jul 20, 2023

I have over 10,000 .blob files, I added them to a .7z (tried to upload before .rar, .zip) archive which ended up being about 40gb in size. Then I tried to upload the archive to the fileshelter, but the process stopped at 100%, I waited all weekend just in case. Further, I thought that the system was trying to get inside the archive, but after making the archive with a password I got the same result. For the sake of interest, I tried to upload .iso file and .bak database backup (each file is 40 GB in size) - it uploaded almost instantly without any problems. At the same time, there are zero errors in the fileshelter.log. Can you help solve the problem?

@epoupon
Copy link
Owner

epoupon commented Jul 20, 2023

Well this is strange.
Fileshelter does not try to open your archive. So uploading a .7z file or a .iso file does not matter and should be the same.

You can try to run with full logs by tweaking this:

log-config = "* -debug -info:WebRequest";

Change to log-config = "*";

@inokaw
Copy link
Author

inokaw commented Jul 24, 2023

corrected the config as you said, in the screenshot the last entries in the log when the upload progress was stuck at 100%
Screenshot_1

@inokaw
Copy link
Author

inokaw commented Jul 27, 2023

@epoupon if you are interested, I can give such an archive by uploading it, for example, to Google Drive

@Silun
Copy link

Silun commented Jan 1, 2024

I am having trouble uploading files as well. I just set it up in docker, and when I upload files, sometimes they just work, other times it just gets stuck at 0% upload:

image

It gets stuck at 0% in about 80-90% of cases larger than about 1 megabyte, I'd say. There is no helpful entry in the logs at all for this, even with the modification you suggested above. The only thing that appears is [info] "WebRequest: took 0.81 ms". I'm not really sure how to proceed. I run fileshelter via docker and behind nginx-proxy. I have tried setting behind-reverse-proxy to true and false without any noticeable difference.

This is my docker-compose:

services:
  fileshelter:
    image: epoupon/fileshelter
    container_name: fileshelter
    user: "1000:1000"
    network_mode: bridge
    restart: always
#    ports:
#      - "5090:5091" #WebUI
    environment:
      - VIRTUAL_PORT=5091
      - VIRTUAL_HOST=fileshelter.my.domain.tld
      - LETSENCRYPT_HOST=fileshelter.my.domain.tld
    volumes:
      - "/home/bsc/fileshelter/var:/var/fileshelter:rw"
      - "/home/bsc/fileshelter/fileshelter.conf:/etc/fileshelter.conf:rw"

and here is my fileshelter.conf:
fileshelter.txt

Edit: This was my fault. I had the client_max_body_size set to the default of 1m. Info on how to change that is available at nginx-proxy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants