-
Notifications
You must be signed in to change notification settings - Fork 5.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support concurrent upload and downloads to same object but different versions #16314
Comments
@ScionOfDesign is this a distributed setup and how big is the set-up? From what I can tell you are concurrently modifying the object as well while downloading? |
Okay I see what you are doing, this is not allowed you cannot add new versions concurrently on top while another version is being downloaded. Versions are a stack and are tied to the object - they do not exist independently. Any concurrent modification of a stack requires serializing them so that reads and writes get consistent results. What you are looking for is not possible and beyond scope of this project at the moment. |
getObjectNInfo() current holds the lock until the client has entirely read the full content from the disk, this is not needed because the dataDir that is present on disk is unique and could never be overwritten concurrently by another caller. It is sufficient to verify xl.meta and its quorum validity under quorum, once that is achieved we do not ever have to hold the lock further. Currently clients speed would dictate the time the lock is held for READ operation on MinIO, this is not needed and we can make this lock-free , allowing mutations to serialize on each other but not READs. This allows for applications like fixes minio#16314
Oh wow, is this going to be addressed after all? Thank you! |
getObjectNInfo() current holds the lock until the client has entirely read the full content from the disk, this is not needed because the dataDir that is present on disk is unique and could never be overwritten concurrently by another caller. It is sufficient to verify xl.meta and its quorum validity under quorum, once that is achieved we do not ever have to hold the lock further. Currently clients speed would dictate the time the lock is held for READ operation on MinIO, this is not needed and we can make this lock-free , allowing mutations to serialize on each other but not READs. This allows for applications like fixes minio#16314
getObjectNInfo() current holds the lock until the client has entirely read the full content from the disk, this is not needed because the dataDir that is present on disk is unique and could never be overwritten concurrently by another caller. It is sufficient to verify xl.meta and its quorum validity under quorum, once that is achieved we do not ever have to hold the lock further. Currently clients speed would dictate the time the lock is held for READ operation on MinIO, this is not needed and we can make this lock-free , allowing mutations to serialize on each other but not READs. This allows for applications like fixes minio#16314
getObjectNInfo() current holds the lock until the client has entirely read the full content from the disk, this is not needed because the dataDir that is present on disk is unique and could never be overwritten concurrently by another caller. It is sufficient to verify xl.meta and its quorum validity under quorum, once that is achieved we do not ever have to hold the lock further. Currently clients speed would dictate the time the lock is held for READ operation on MinIO, this is not needed and we can make this lock-free , allowing mutations to serialize on each other but not READs. This allows for applications like fixes minio#16314
@ScionOfDesign Yes. But it has to be a safe change, which will require fundamental changes to locking. |
So, while there is someone downloading a file, there is no way deleting it or creating a new version (overriding it)? That's awful! Imagine someone "locking" all your objects by downloading them very slowly all day, every day! 😞 create and upload a file: dd if=/dev/urandom of=64MB.bin bs=64M count=1 iflag=fullblock
mc cp ./64MB.bin testminio/testbucket/ in a different terminal start downloading the file: wget --limit-rate=50k -O /dev/null https://testminio/testbucket/64MB.bin in a different terminal, while the file is being downloaded both of these commands would "hang": mc cp ./64MB.bin testminio/testbucket/
mc rm testminio/testbucket/64MB.bin |
@taai, you know a fix; feel free to send it. Go ahead and write a distributed locking that is strictly consistent and safe.
They are not hung; they simply wait for the lock to be relinquished to perform the DELETE call. Just so you know, we have already addressed this for small objects. However, we are working on addressing this comprehensively overall by using a different kind of sharded locking implementation. One of the reasons why I reopened this GitHub, hold your horses while we get this out. |
Expected Behavior
When uploading a new version of a versioned file, I would expect it to be able to be uploaded even if users are downloading the previous version.
Current Behavior
When uploading a new version of a file that is heavily downloaded, I get the following error:
mc: <ERROR> Failed to copy `MyFile.zip`. A timeout occurred while trying to lock a resource, please reduce your request rate
Possible Solution
Place a CDN in front of our MinIO server. Probably a preferable solution, but I don't understand why MinIO doesn't handle this situation as well.
Steps to Reproduce (for bugs)
minio cp
a new version on top of the only version.Context
We make games with releases stored on MinIO, and occasionally release large updates. These updates lead to a spike in traffic to our MinIO server.
However, if we need to upload a hotfix, we need to restart the server or otherwise sever the active connections in order to be able to upload a new neversion. Is it possible to only have old
GET
requests continue downloading the old file until the new file is uploaded, (in which case the older requests would still download the old version) and have all new requests download the new file?Regression
No
Your Environment
Minio instance installed with helm and the MInio Operator, running on Kubernetes. Traefik for routing.
Version
RELEASE.2022-11-11T03-44-20Z
.The text was updated successfully, but these errors were encountered: