taucmdr.cf.storage package

TAU Storage Containers.

A storage container provides a record database, a persistent filesystem, and a key/value store. The database is accessed via member methods like AbstractStorage.insert(), AbstractStorage.search(), etc. The filesystem is accessed via its filesystem prefix (e.g. /usr/local/packages) via AbstractStorage.prefix(). The key/value store is accessed via the [] operator, i.e. treat the storage object like a dictionary.

class taucmdr.cf.storage.AbstractStorage(name: str)

Bases: object

Abstract base class for storage containers.

A storage container provides a record database, a persistent filesystem, and a key/value store. The database is accessed via member methods like insert(), search(), etc. The filesystem is accessed via its filesystem prefix (e.g. /usr/local/packages) via prefix(). The key/value store is accessed via the [] operator, i.e. treat the storage object like a dictionary.

name

The storage container’s name, e.g. “system” or “user”.

Type:

str

Record

alias of StorageRecord

abstractmethod __contains__(key)

Returns True if key maps to a value is in the key/value store.

abstractmethod __delitem__(key)

Remove a value from the key/value store.

abstractmethod __enter__()

Initiates the database transaction.

abstractmethod __exit__(ex_type, value, traceback)

Finalizes the database transaction.

abstractmethod __getitem__(key)

Retrieve a value from the key/value store.

abstractmethod __iter__()

Iterate over keys in the key/value store.

abstractmethod __len__()

Return the number of items in the key/value store.

abstractmethod __setitem__(key, value)

Store a value in the key/value store.

_abc_impl = <_abc._abc_data object>

abstractmethod connect_database(*args, **kwargs)

Open the database for reading and writing.

abstractmethod connect_filesystem(*args, **kwargs)

Prepares the store filesystem for reading and writing.

abstractmethod contains(keys, table_name=None, match_any=False)

Check if the specified table contains at least one matching record.

The behavior depends on the type of keys:

• self.Record.eid_type: check for the record with that element identifier.

• dict: check for the record with attributes matching keys.

• list or tuple: return the equivalent of map(contains, keys).

Parameters:

• keys – Fields or element identifiers to match.

• table_name (str) – Name of the table to operate on. See AbstractStorage.table.

• match_any (bool) – Only applies if keys is a dictionary. If True then any key in keys may match or if False then all keys in keys must match.

Returns:

True if the table contains at least one matching record, False otherwise.

Return type:

bool

Raises:

ValueError – Invalid value for keys.

abstractmethod count(table_name=None)

Count the records in the database.

Parameters:

table_name (str) – Name of the table to operate on. See AbstractStorage.table.

Returns:

Number of records in the table.

Return type:

int

abstractmethod database_exists()

Determine if the database file backing this Storage object exists.

Returns:

True if database exists; false otherwise.

Return type:

bool

abstractmethod disconnect_database(*args, **kwargs)

Close the database for reading and writing.

abstractmethod disconnect_filesystem(*args, **kwargs)

Makes the store filesystem unreadable and unwritable.

abstractmethod get(keys, table_name=None, match_any=False)

Find a single record.

The behavior depends on the type of keys:

• self.Record.eid_type: return the record with that element identifier.

• dict: return the record with attributes matching keys.

• list or tuple: return a list of records matching the elements of keys

• None: return None.

Parameters:

• keys – Fields or element identifiers to match.

• table_name (str) – Name of the table to operate on. See AbstractStorage.table.

• match_any (bool) – Only applies if keys is a dictionary. If True then any key in keys may match or if False then all keys in keys must match.

Returns:

The matching data record if keys was a self.Record.eid_type or dict. list: All matching data records if keys was a list or tuple. None: No record found or bool(keys) == False.

Return type:

Record

Raises:

ValueError – Invalid value for keys.

abstractmethod insert(data, table_name=None)

Create a new record.

If the table doesn’t exist it will be created.

Parameters:

• data (dict) – Data to insert in table.

• table_name (str) – Name of the table to operate on. See AbstractStorage.table.

Returns:

The new record.

Return type:

Record

abstractmethod items()

Iterate over items in the key/value store.

abstractmethod keys()

Iterate over keys in the key/value store.

abstractmethod match(field, table_name=None, regex=None, test=None)

Find records where field matches regex or test.

Either regex or test may be specified, not both. If regex is given, then all records with field matching the regular expression are returned. If test is given then all records with field set to a value that causes test to return True are returned. If neither is given, return all records where field is set to any value.

Parameters:

• table_name (str) – Name of the table to operate on. See AbstractStorage.table.

• field (string) – Name of the data field to match.

• regex (string) – Regular expression string.

• test – Callable returning a boolean value.

Returns:

Matching data records.

Return type:

list

Raises:

ValueError – Invalid value for keys.

abstract property prefix: str

Get the filesystem prefix for file storage.

The filesystem must be persistent and provide the usual POSIX filesystem calls. In particular, GNU software packages should be installable in the filesystem.

Returns:

Absolute path in the filesystem.

Return type:

str

abstractmethod purge(table_name=None)

Delete all records.

Parameters:

table_name (str) – Name of the table to operate on. See AbstractStorage.table.

abstractmethod remove(keys, table_name=None, match_any=False)

Delete records.

The behavior depends on the type of keys:

• self.Record.eid_type: delete the record with that element identifier.

• dict: delete all records with attributes matching keys.

• list or tuple: delete all records matching the elements of keys.

Parameters:

• keys – Fields or element identifiers to match.

• table_name (str) – Name of the table to operate on. See AbstractStorage.table.

• match_any (bool) – Only applies if keys is a dictionary. If True then any key in keys may match or if False then all keys in keys must match.

Raises:

ValueError – bool(keys) == False or invalid value for keys.

abstractmethod search(keys=None, table_name=None, match_any=False)

Find multiple records.

The behavior depends on the type of keys:

• Record.eid_type: return the record with that element identifier.

• dict: return all records with attributes matching keys.

• list or tuple: return a list of records matching the elements of keys

• None: return all records.

Parameters:

• keys – Fields or element identifiers to match.

• table_name (str) – Name of the table to operate on. See AbstractStorage.table.

• match_any (bool) – Only applies if keys is a dictionary. If True then any key in keys may match or if False then all keys in keys must match.

Returns:

Matching data records.

Return type:

list

Raises:

ValueError – Invalid value for keys.

abstractmethod table(table_name)

Return a handle to a table.

Return a handle to the named table or, if table_name is None, return the default table.

Parameters:

table_name (str) – Name of the table or None.

Returns:

A database table object.

Return type:

object

abstractmethod unset(fields, keys, table_name=None, match_any=False)

Update records by unsetting fields.

Update only allows you to update a record by adding new fields or overwriting existing fields. Use this method to remove a field from the record.

The behavior depends on the type of keys:

• self.Record.eid_type: update the record with that element identifier.

• dict: update all records with attributes matching keys.

• list or tuple: apply update to all records matching the elements of keys.

Parameters:

• fields (list) – Names of fields to remove from matching records.

• keys – Fields or element identifiers to match.

• table_name (str) – Name of the table to operate on. See AbstractStorage.table.

• match_any (bool) – Only applies if keys is a dictionary. If True then any key in keys may match or if False then all keys in keys must match.

Raises:

ValueError – bool(keys) == False or invalid value for keys.

abstractmethod update(fields, keys, table_name=None, match_any=False)

Update records.

The behavior depends on the type of keys:

• self.Record.eid_type: update the record with that element identifier.

• dict: update all records with attributes matching keys.

• list or tuple: apply update to all records matching the elements of keys.

Parameters:

• fields (dict) – Data to record.

• keys – Fields or element identifiers to match.

• table_name (str) – Name of the table to operate on. See AbstractStorage.table.

• match_any (bool) – Only applies if keys is a dictionary. If True then any key in keys may match or if False then all keys in keys must match.

Raises:

ValueError – bool(keys) == False or invalid value for keys.

abstractmethod values()

Iterate over values in the key/value store.

exception taucmdr.cf.storage.StorageError(value, *hints)

Bases: Error

Indicates a failure in the storage system.

message_fmt = '%(value)s\n%(hints)s\n'

Format string for the error message.

class taucmdr.cf.storage.StorageRecord(storage, eid, element)

Bases: dict

A record in the storage container’s database.

storage

Storage container whose database contains this record.

eid

Element identifier value.

eid_type = None

Element identifier type.

Subpackages

• taucmdr.cf.storage.tests package
  • Submodules
    • taucmdr.cf.storage.tests.test_levels module
      • LevelsTest
    • taucmdr.cf.storage.tests.test_local_file module
      • LocalFileTest
    • taucmdr.cf.storage.tests.test_old_databases module
      • OldDatabaseTest
    • taucmdr.cf.storage.tests.test_project module
      • ProjectTest
    • taucmdr.cf.storage.tests.test_sqlite3_file module
      • SQLite3FileStorageTests
    • taucmdr.cf.storage.tests.test_sqlite3_project module
      • Sqlite3ProjectTest
    • taucmdr.cf.storage.tests.test_storage_dispatch module
      • StorageDispatchTests

Submodules

• taucmdr.cf.storage.levels module
  • ORDERED_LEVELS
  • PROJECT_STORAGE
  • STORAGE_LEVELS
  • SYSTEM_STORAGE
  • USER_STORAGE
  • build_level()
  • highest_writable_storage()
  • require_system_storage()
  • search_levels()
  • writable_system_storage()
• taucmdr.cf.storage.local_file module
  • LocalFileStorage
    • LocalFileStorage.Record
    • LocalFileStorage.__enter__()
    • LocalFileStorage.__exit__()
    • LocalFileStorage.__str__()
    • LocalFileStorage._abc_impl
    • LocalFileStorage._make_prefix()
    • LocalFileStorage._query()
    • LocalFileStorage.connect_database()
    • LocalFileStorage.connect_filesystem()
    • LocalFileStorage.contains()
    • LocalFileStorage.count()
    • LocalFileStorage.database_exists()
    • LocalFileStorage.dbfile
    • LocalFileStorage.disconnect_database()
    • LocalFileStorage.disconnect_filesystem()
    • LocalFileStorage.get()
    • LocalFileStorage.insert()
    • LocalFileStorage.is_writable()
    • LocalFileStorage.items()
    • LocalFileStorage.keys()
    • LocalFileStorage.match()
    • LocalFileStorage.prefix
    • LocalFileStorage.purge()
    • LocalFileStorage.remove()
    • LocalFileStorage.search()
    • LocalFileStorage.table()
    • LocalFileStorage.unset()
    • LocalFileStorage.update()
    • LocalFileStorage.values()
  • _CachingMiddleware
    • _CachingMiddleware.flush()
  • _JsonFileStorage
    • _JsonFileStorage._abc_impl
    • _JsonFileStorage.read()
    • _JsonFileStorage.write()
  • _JsonRecord
    • _JsonRecord.eid_type
• taucmdr.cf.storage.project module
  • ProjectStorage
    • ProjectStorage._abc_impl
    • ProjectStorage.connect_filesystem()
    • ProjectStorage.destroy()
    • ProjectStorage.force_cwd()
    • ProjectStorage.prefix
    • ProjectStorage.tau_dir()
  • ProjectStorageError
    • ProjectStorageError.message_fmt
• taucmdr.cf.storage.sqlite3_file module
  • SQLiteDatabase
    • SQLiteDatabase.dbfile
    • SQLiteDatabase._LoggingCursor
      • SQLiteDatabase._LoggingCursor.close()
      • SQLiteDatabase._LoggingCursor.execute()
      • SQLiteDatabase._LoggingCursor.fetchall()
      • SQLiteDatabase._LoggingCursor.fetchone()
      • SQLiteDatabase._LoggingCursor.lastrowid
    • SQLiteDatabase.check_writable()
    • SQLiteDatabase.close()
    • SQLiteDatabase.commit_transaction()
    • SQLiteDatabase.cursor()
    • SQLiteDatabase.open()
    • SQLiteDatabase.purge()
    • SQLiteDatabase.revert_transaction()
    • SQLiteDatabase.start_transaction()
    • SQLiteDatabase.table()
    • SQLiteDatabase.tables()
  • SQLiteLocalFileStorage
    • SQLiteLocalFileStorage.Record
    • SQLiteLocalFileStorage.__enter__()
    • SQLiteLocalFileStorage.__exit__()
    • SQLiteLocalFileStorage.__str__()
    • SQLiteLocalFileStorage._abc_impl
    • SQLiteLocalFileStorage._database_writable()
    • SQLiteLocalFileStorage.connect_database()
    • SQLiteLocalFileStorage.contains()
    • SQLiteLocalFileStorage.count()
    • SQLiteLocalFileStorage.database_exists()
    • SQLiteLocalFileStorage.dbfile
    • SQLiteLocalFileStorage.disconnect_database()
    • SQLiteLocalFileStorage.get()
    • SQLiteLocalFileStorage.insert()
    • SQLiteLocalFileStorage.match()
    • SQLiteLocalFileStorage.purge()
    • SQLiteLocalFileStorage.remove()
    • SQLiteLocalFileStorage.search()
    • SQLiteLocalFileStorage.table()
    • SQLiteLocalFileStorage.unset()
    • SQLiteLocalFileStorage.update()
  • SQLiteStorageError
    • SQLiteStorageError.message_fmt
  • _SQLiteJsonRecord
    • _SQLiteJsonRecord.eid_type
  • _SQLiteJsonTable
    • _SQLiteJsonTable.Record
    • _SQLiteJsonTable._ensure_exists()
    • _SQLiteJsonTable._get()
    • _SQLiteJsonTable._json_query()
    • _SQLiteJsonTable.count()
    • _SQLiteJsonTable.exists()
    • _SQLiteJsonTable.get()
    • _SQLiteJsonTable.insert()
    • _SQLiteJsonTable.name
    • _SQLiteJsonTable.purge()
    • _SQLiteJsonTable.remove()
    • _SQLiteJsonTable.search()
    • _SQLiteJsonTable.update()
• taucmdr.cf.storage.sqlite3_project module
  • SQLiteProjectStorage
    • SQLiteProjectStorage._abc_impl
    • SQLiteProjectStorage.connect_filesystem()
    • SQLiteProjectStorage.destroy()
    • SQLiteProjectStorage.force_cwd()
    • SQLiteProjectStorage.prefix
    • SQLiteProjectStorage.tau_dir()
• taucmdr.cf.storage.storage_dispatch module
  • ProjectStorageDispatch
    • ProjectStorageDispatch._abc_impl
    • ProjectStorageDispatch.destroy()
  • StorageDispatch
    • StorageDispatch.__contains__()
    • StorageDispatch.__delitem__()
    • StorageDispatch.__enter__()
    • StorageDispatch.__exit__()
    • StorageDispatch.__getattr__()
    • StorageDispatch.__getitem__()
    • StorageDispatch.__iter__()
    • StorageDispatch.__len__()
    • StorageDispatch.__setitem__()
    • StorageDispatch._abc_impl
    • StorageDispatch._get_storage()
    • StorageDispatch.connect_database()
    • StorageDispatch.connect_filesystem()
    • StorageDispatch.contains()
    • StorageDispatch.count()
    • StorageDispatch.database_exists()
    • StorageDispatch.default_backend
    • StorageDispatch.disconnect_database()
    • StorageDispatch.disconnect_filesystem()
    • StorageDispatch.get()
    • StorageDispatch.insert()
    • StorageDispatch.items()
    • StorageDispatch.keys()
    • StorageDispatch.match()
    • StorageDispatch.prefix
    • StorageDispatch.purge()
    • StorageDispatch.remove()
    • StorageDispatch.search()
    • StorageDispatch.set_backend()
    • StorageDispatch.table()
    • StorageDispatch.unset()
    • StorageDispatch.update()
    • StorageDispatch.values()