filesystem
- PyFilesystem based Storage¶
The filesystem
backend can use any filesystem-like storage that is
supported by PyFilesystem.
Local Filesystem Storage¶
Most notably, the filesystem
backend can be used to store metadata in, well,
your local file system. This is typically useful in very small, PoC or test
setups or for automated testing where storage state needs to be preserved
between isolated tests (and the mem://
backend will not suffice).
To store metadata locally, simply instantiate using the filesystem
backend
and the "uri"
option pointing to a local directory:
import metastore
backend = metastore.create_metastore('filesystem', {"uri": "./metastore"})
Note that the local directlry (in our case ./metastore
relative to the
current working directory) must exist before backend
can be instantiated
and used.
In-Memory Storage for Testing Purposes¶
Another potential use of this backend is when writing tests for code that uses
metastore-lib
. instead of mocking out metastore-lib
code, one can simply
use the mem
filesystem when instantiating a metastore-lib
backend instance:
import metastore
backend = metastore.create_metastore('filesystem', {"uri": "mem://"})
This backend object will only store metadata in the process memory, and only
for as long as the backend
object is alive. Once the object is destroyed
(e.g. at the end of a test), all data stored in it will be destroyed as well.
This comes in handy in testing because it is a super-fast, limited state
storage, which doesn’t require any special cleanup procedures between tests.
Other Filesystem-Like Backends¶
PyFilesystem abstracts a large number of filesystem like storage backends,
and all of them can be used with metastore-lib
assuming you have the proper
dependencies installed.
All you need to do is provide the right
PyFilesystem URL as
the "uri"
configuration option when instantiating the backend.
See https://docs.pyfilesystem.org/en/latest/builtin.html for a full list of built-in supported filysystems, and https://docs.pyfilesystem.org/en/latest/implementers.html for details on how to implement your own filesystem abstraction.