Better management of automatic cloud vs. custom data links
planned
C
Coral reef Lamprey
We have a relatively complex Seqera Platform/Tower installation and manage our system via the API. While there are places we'd like to provide people the ability to browse specific buckets within SP/Tower (Task workspace buckets, input & output buckets, and some other niceties), we definitely don't want to provide open access (including uploads) to every bucket available within our workspaces.
We're exploring what's in the UI, what's in the configuration, and what's in the API & DB, and we're finding some pretty weird things re: our ability to automatically / programmatically manage which buckets show up in the data explorer.
We'd really like to encourage maybe flattening the overall product surface here so there's not a bifurcation between what we add and what SP/Tower automatically adds based on its own logic, and also maybe some additional tunables around which buckets can be uploaded to, etc.
The data explorer is great, the magic parts are not.
Rob Newman
planned
Rob Newman
Adding functionality to disable or enable automatic cloud data-links per workspace via Workspace settings (or via an API endpoint) is in the Engineering backlog.
Rob Newman
Thanks for the feedback, Eric. Data Explorer is still in public-preview and we're actively improving and standardizing the UI and API to be consistent so your input is very valuable.
Currently, only workspace users with the
Maintain
role and above can upload, download and preview files in the Data Explorer, so you can use role assignments to determine which of your users can access cloud storage buckets attached to your workspaces.Data Explorer currently lists all buckets accessible to your workspace cloud credentials. To make this more limited across a workspace, you can modify your workspace credentials to be more strict, and then have a
Maintain
role+ workspace user select the Add cloud bucket
feature and manually manage cloud storage buckets and custom data-links. (Note that you can also do this programmatically via a series of GET/POST/PUT/DELETE
API requests to the data-links
endpoint but I'm aware that we need to update our API docs to include this endpoint)Configuration-wise, you can disable automatic cloud bucket retrieval per workspace by using comma-separated workspace IDs in the
TOWER_DATA_EXPLORER_CLOUD_DISABLED_WORKSPACES
environment variable or defined in tower.yml
file. Docs.I think it would be helpful for me to better understand the "weird things" your team are encountering by jumping on a call. I'll reach out via email today.
Rob Newman
acknowledged