Georgehwp is looking for a solution to handle large datasets in computer vision projects with data that changes frequently, such as daily appends. Romil Bhardwaj suggests using SkyPilot's in-built bucket mounting feature for streaming data from the bucket. Additionally, Romil mentions the tradeoffs in performance compared to using s5cmd. Georgehwp also shares his experience with s5cmd, stating that it was faster than the default usage of the AWS S3 CLI.
Georgehwp
Asked on Feb 19, 2024