Add remote storage#

Hide code cell content
!lamin close
!yes | lamin delete pgtest
import lamindb_setup as ln_setup
from lamindb_setup._add_remote_storage import switch_default_storage

Add remote storage#

Hide code cell content
import laminci

!docker stop pgtest && docker rm pgtest
pgurl = laminci.db.setup_local_test_postgres()
ln_setup.init(storage="./storage1", db=pgurl)
ln_setup.settings.storage.root

Local storage:

switch_default_storage("./storage_2")
ln_setup.settings.storage.root
ln_setup.settings.storage.root_as_str
Hide code cell content
from pathlib import Path

assert ln_setup.settings.storage.root_as_str == f"{Path.cwd()}/storage_2"

Cloud storage:

switch_default_storage("s3://lamindb-ci")
ln_setup.settings.storage.root

See an overview:

Hide code cell content
assert ln_setup.settings.storage.is_cloud
assert ln_setup.settings.storage.root_as_str == "s3://lamindb-ci"
assert ln_setup.settings.storage.region == "us-west-1"
# root.fs contains the underlying fsspec filesystem
assert (
    ln_setup.settings.storage.root.fs.cache_regions  # set by lamindb to True for s3 by default
)

You can set any additional fsspec filesystem arguments for cloud storage, such as profile or cache_regions (for s3 only), for example:

switch_default_storage("s3://lamindb-ci", cache_regions=False)
Hide code cell content
# test cache_regions
assert not ln_setup.settings.storage.root.fs.cache_regions
# test setting storage not by owner
ln_setup.login("testuser2@lamin.ai", key="goeoNJKE61ygbz1vhaCVynGERaRrlviPBVQsjkhz")
switch_default_storage("./storage_3")
assert ln_setup.settings.storage.root_as_str == f"{Path.cwd()}/storage_3"
!docker stop pgtest && docker rm pgtest

Currently not possible: setting storage for SQLite instance#

If you try to set the storage for an sqlite instance, an error message is returned:

assert switch_default_storage("mydata_storage_2") == "set-storage-failed"