Working with scripts¶

$ om scripts -h
Usage:
 om scripts list [<pattern>] [--raw] [--hidden] [-E|--regexp] [options]
 om scripts put <path> <name> [options]
 om scripts get <name>
 om scripts drop <name> [options]
 om scripts metadata <name>

Note

How scripts work

omegaml scripts are simply pip-installable modules that expose a run() function returning a pickle-serializable result. omegaml works like a thin wrapper around the Python standard tools setuptools and pip modules:

Upon storing a script, omegaml creates a package and stores it its distributed filesystem.
Upon retrieving a script, the package is installed. Packages are created using python setup.py sdist. Packages are installed using pip install <package>.
In case of remotely hosted packages, omegaml stores the reference to the remote location. In this case python setup.py sdist is not run. If you have stored a local package and wish to replace it by a remote reference, drop the package first as local and remote packages result in different Metadata kind types, respectively.

Packaging local modules¶

Scripts are pip-installable packages. To package-up a script and store it, specify the <path> to the setup.py:

$ om scripts put process/setup.py process
running sdist
running check
warning: sdist: manifest template 'MANIFEST.in' does not exist (using default file list)

writing manifest file 'MANIFEST'
creating process-1.3
creating process-1.3/process
making hard links in process-1.3...
hard linking README -> process-1.3
hard linking setup.py -> process-1.3
hard linking process/__init__.py -> process-1.3/process
Creating tar archive
removing 'process-1.3' (and everything under it)
Metadata(name=process,bucket=omegaml,prefix=scripts/,kind=python.package,created=2020-08-28 13:58:32.407000)

Note

The name of the package created must be the importable module name
The name of the package must contain alphanumeric characters only
Scripts can import other modules by specifying dependencies. See install_requires in the Python packaging guide

In this example, the following directory structure, setup.py and code was used:

# structure
./setup.py
./process/__init_.py

# setup.py
#!/usr/bin/env python
from distutils.core import setup
setup(name='process',
      version='1.3',
      description='simple omegaml hello world script',
      author='omegaml',
      author_email='info@omegaml.io',
      url='http://omegaml.io',
      packages=['process'])

#__init__.py
def run(om, *args, **kwargs):
    return "hello world"

Packaging pypi-hosted modules¶

Instead of building and packaging modules it is possible to store a reference to a specific Pypi-hosted module:

$ om scripts put "six==1.0" six

Upon installation, the script will be pulled from pypi and installed as usual:

$ om scripts get six
Collecting six==1.0
  Downloading six-1.0.0.tar.gz (11 kB)
Building wheels for collected packages: six
  Building wheel for six (setup.py) ... done
  Created wheel for six: filename=six-1.0.0-py3-none-any.whl size=4871 sha256=c966e8ada020a84af438ca7a1baf0e0597200d84f82f704425a4a00885b29e66
  Stored in directory: /tmp/pip-ephem-wheel-cache-3f58ar72/wheels/c5/45/ed/be3d9e59c2233eb179a23b38d5a74b20b139c689fcf16f9ca5
Successfully built six
Installing collected packages: six
Successfully installed six-1.0.0

Packaging git-hosted modules¶

Instead of building and packaging modules it is possible to store references to git-hosted modules, using the pip vcs support

$ om scripts put "git+https://github.com/omegaml/apps.git#egg=helloworld&subdirectory=helloworld" helloworld

Upon installation, the script will be cloned from the git repository and installed as usual:

$ om scrpits get helloworld
Collecting helloworld
  Cloning https://github.com/omegaml/apps.git to /tmp/pip-install-yn92_8a9/helloworld_20e05893ad1d4e92bafbc281eb121236
  Running command git clone -q https://github.com/omegaml/apps.git /tmp/pip-install-yn92_8a9/helloworld_20e05893ad1d4e92bafbc281eb121236
Building wheels for collected packages: helloworld
  Building wheel for helloworld (setup.py) ... done
  Created wheel for helloworld: filename=helloworld-1.0-py3-none-any.whl size=2192 sha256=d8b8cac1ad5e8a1c3f9dcc46cbf10634acb2f7e09e81ffc9ff0f6c38d0f08219
  Stored in directory: /tmp/pip-ephem-wheel-cache-p9kxp493/wheels/3d/10/b9/be33eac8519a9ae4dd329947255f240b5ccde8d01554155f5e
Successfully built helloworld
Installing collected packages: helloworld
Successfully installed helloworld-1.0

Installing modules¶

A module can be installed as follows:

$ om scripts get helloworld
...
Successfully installed helloworld-1.0

Running scripts remotely¶

Scripts are executed by the runtime as follows

$ om runtime script helloworld
{"script": "helloworld", "kwargs": {"pure_python": false}, "result": "hello world", "runtimes": 0.090067, "started": "2021-02-20T16:30:35.837488", "ended": "2021-02-20T16:30:35.927555"}

Running scripts locally¶

Scripts can be executed locally instead of the the runtime by specifying the --local flag. Use this for debugging:

$ om runtime script helloworld --local
...
Successfully installed helloworld-1.3
Task omegaml.backends.package.tasks.run_omega_script[329e6fb1-d2d6-4a24-881d-b17c9c917910] succeeded in 4.052356380969286s: '{"script": "process", "kwargs": {"pure_python": false}, "result": "Metadata(name=model_results,bucket=omegaml,prefix=data/,kind=python.data,created=2021-02-11 08:40:38.655000)", "runtimes": 0.308365, "started": "2021-02-20T17:34:33.057252", "ended": "2021-02-20T17:34:33.365617"}'
{"script": "helloworld", "kwargs": {"pure_python": false}, "result": "hello world", "runtimes": 0.090067, "started": "2021-02-20T16:30:35.837488", "ended": "2021-02-20T16:30:35.927555"}

Note

omegaml leverages Celery for local and remote executing of scripts:

Ask Celery to run a task (“execute the run_omega_script task for script “helloworld”)
Celery receives the message, starts the corresponding task
om.scripts.get() installs the script in the runtime’s environment (local or remote)
import helloworlds imports the module
call helloworld.run(...) and return its results

To learn about Celery and its distributed task model see Celery