om.scripts

omegaml.scripts = OmegaStore(bucket=omegaml, prefix=scripts/)

the omegaml.store.base.OmegaStore store for scripts

Methods:

Mixins:

Backends:

Backends

class omegaml.backends.package.PythonPackageData(model_store=None, data_store=None, tracking=None, **kwargs)

Backend to support locally sourced custom scripts deployment to runtimes cluster

This supports any local setup.py

Usage:

om.scripts.put('pkg://path/to/setup.py', 'myname')
om.scripts.get('myname')

Note that setup.py must minimally specify the following. The
name and packages kwargs must specify the same name as given
in put(..., name)

    # setup.py
    from setuptools import setup
    setup(name='myname', packages=['myname'])

See also

https://packaging.python.org/tutorials/packaging-projects/

KIND = 'python.package'
get(name, localpath=None, keep=False, install=True, **kwargs)

Load package from store, install it locally and load.

This copies the package’s .tar.gz file from om.scripts to a local temp path and runs pip install on it.

Parameters:

  • name – the name of the package

  • keep – keep the packages load path in sys.path, defaults to False

  • localpath – the local path to store the package

  • install – if True call pip install on the retrieved package

  • kwargs

Returns:

the loaded module

put(obj, name, attributes=None, **kwargs)

save a python package

This takes the full path to a setuptools setup.py, or a directory containing a setup.py file. It then executes python setup.py sdist and stores the resulting .tar.gz file in om.scripts

Parameters:

  • obj – full path to package file or directory, syntax as pkg://path/to/dist.tar.gz or pkg://path/to/setup.py

  • name – name of package. must be the actual name of the package as specified in setup.py

Returns:

the Metadata object

classmethod supports(obj, name, **kwargs)

test if this backend supports this obj

class omegaml.backends.package.PythonPipSourcedPackageData(model_store=None, data_store=None, tracking=None, **kwargs)

Backend to support locally sourced custom scripts deployment to runtimes cluster

This supports any pip-installable remote source like git, hg, svn, bzr

Usage:

# pypi package
om.scripts.put('pypi://<pkgname>', '<pkgname>')

# git hosted
om.scripts.put('git+https://github.com/account/repo/<pkgname>', '<pkgname>')

# retrieve the package and call pip install
om.scripts.get('name')

Notes

  • pypi://<pkgname> can be any specification supported by pip, e.g.:

    om.scripts.put('pypi://<pkgname==version>', ...)
om.scripts.put('git+https://....@tag#egg=<pkgname>', ...)

  • the package specification is stored as metadata.kind_meta['pip_source']

See also

KIND = 'pipsrc.package'
get(name, keep=False, **kwargs)

Load package from store, install it locally and load.

This runs pip install <package spec>, taking the package specification from Metadata.kind_meta[‘pip-source’]. See the put() method for details.

Parameters:

  • name – the name of the package

  • keep – keep the packages load path in sys.path, defaults to False

  • kwargs

Returns:

the loaded module

put(obj, name, attributes=None, **kwargs)

save a python package

This stores a PIP-sourceable package specification in the objects Metadata.kind_meta:

{ 'pip_source': '<obj:package spec>' }
Parameters:

  • obj – full path to package file or directory, syntax as pkg://path/to/dist.tar.gz or pkg://path/to/setup.py

  • name – name of package. must be the actual name of the package as specified in setup.py

Returns:

the Metadata object

classmethod supports(obj, name, **kwargs)

test if this backend supports this obj

class omegaml.backends.mlflow.localprojects.MLFlowProjectBackend(model_store=None, data_store=None, tracking=None, **kwargs)

Backend to support storage of MLFlow projects

Usage:

om.scripts.put('mlflow://path/to/MLProject', 'myproject')
om.scripts.get('myproject')

See also

https://www.mlflow.org/docs/latest/projects.html#project-directories

KIND = 'mlflow.project'
get(name, localpath=None, **kwargs)

Load MLFlow project from store

This copies the projects’s .tar.gz file from om.scripts to a local temp path and returns the MLFlowProject to it

Parameters:

  • name – the name of the package

  • keep – keep the packages load path in sys.path, defaults to False

  • localpath – the local path to store the package

  • install – if True call pip install on the retrieved package

  • kwargs

Returns:

the loaded module

put(obj, name, attributes=None, **kwargs)

save a MLFlow project

Parameters:

  • obj – full path to the MLFlow project directory mlflow://path/to/MLProject

  • name – name to store

Returns:

the Metadata object

classmethod supports(obj, name, **kwargs)

test if this backend supports this obj

class omegaml.backends.mlflow.gitprojects.MLFlowGitProjectBackend(model_store=None, data_store=None, tracking=None, **kwargs)

Backend to support git-sourced MLFlow projects

This supports any git remote source that MLFlow supports

Usage:

om.scripts.put('mlflow+ssh://git@github.com/<user>/<repo>.git', '<pkgname>')
om.scripts.put('mlflow+https://git@github.com/<user>/<repo>.git', '<pkgname>')
om.scripts.put('https://github.com/<user>/<repo>', '<pkgname>', kind='mlflow.project')

See also

KIND = 'mlflow.gitproject'
get(name, **kwargs)

Load MLFlow project from git uri

Returns:

the loaded module

put(obj, name, attributes=None, **kwargs)

save a MLFlow git-sourceable project

classmethod supports(obj, name, **kwargs)

test if this backend supports this obj

class omegaml.backends.rsystem.rscripts.RPackageData(model_store=None, data_store=None, tracking=None, **kwargs)

Backend to support R packages

Usage:

om.scripts.put('R://path/to/app.R', 'myname')
om.scripts.get('myname')

See also

KIND = 'package.r'
get(name, localpath=None, keep=False, install=True, **kwargs)

Load package from store, install it locally and load.

This copies the package’s .tar.gz file from om.scripts to a local temp path and runs pip install on it.

Parameters:

  • name – the name of the package

  • keep – keep the packages load path in sys.path, defaults to False

  • localpath – the local path to store the package

  • install – if True call pip install on the retrieved package

  • kwargs

Returns:

the loaded module (RScript)

put(obj, name, attributes=None, **kwargs)

save a R package

This takes the full path to a setuptools setup.py, or a directory containing a setup.py file. It then executes python setup.py sdist and stores the resulting .tar.gz file in om.scripts

Parameters:

  • obj – full path to package file or directory, syntax as pkg://path/to/dist.tar.gz or pkg://path/to/setup.py

  • name – name of package. must be the actual name of the package as specified in setup.py

Returns:

the Metadata object

classmethod supports(obj, name, **kwargs)

test if this backend supports this obj

Mixins

class omegaml.mixins.store.package.PythonPackageMixin

Install and load scripts

install(specs=None, keep=False)

install and load packages

This is a convenience function to install one or more packages at once. It is the equivalent of

for pkg in [‘name1’, ‘name2’]:

om.scripts.get(pkg)

Parameters:

  • specs (str, list) – optional, package name or list of names, defaults to self.list()

  • keep (bool) – optional, will keep any packages installed in sys.path, defaults to False

Helpers

class omegaml.backends.rsystem.rscripts.RScript(appdir)

a Python proxy to the R process that runs a script

This provides the mod.run() interface for scripts so that we can use the same semantics for R and python scripts.

run(om, **kwargs)

run the script in R session

Usage:

The script must exist as {self.appdir}/app.R. It must implement the omega_run() function.

Example:

# app.R
library(jsonlite)
omega_run <- function(om, kwargs) {
   # if om was passed as "0", this means we're in a local mode, i.e. must import omegaml
   om <- if (om == "0") import("omegaml") else om
   s <- fromJSON(rawToChar(base64_dec(kwargs)))
   s$message <- "hello from R"
   return(toJSON(s))
}

Notes

  • If an R session is active, will run the script by calling the script’s omega_run function

  • If no R session is active, will use RScript to source the script and run omega_run function

  • Expects the output to be in JSON format