Experiment Tracking¶
To set up experiment tracking, we specify the experiment as a context to the runtime:
# assuming we have a model lr and data X and Y
with om.runtime.experiment('myexp') as exp:
score = lr.score(X, Y)
exp.log_metric('accuracy', score)
To retrieve the experiment’s data, we can retrieve back the experiment data, as a dataframe:
exp = om.runtime.experiment('myexp')
data = exp.data()
| | experiment | run | step | event | key | value | dt | node | userid |
|----|--------------|-------|--------|---------|-------|---------|----------------------------|--------|----------|
| 1 | iris | 3 | | start | start | nan | 2024-06-22 17:00:42.862633 | helio | patrick |
| 0 | iris | 3 | | metric | score | 0.5 | 2024-06-22 17:00:42.867853 | helio | patrick |
| 2 | iris | 3 | | stop | stop | nan | 2024-06-22 17:00:42.871592 | helio | patrick |
Note that the experiment automatically captured start and stop events, in addition to the actual metric logged. This is done automatically by the experiment context manager. The start event is logged at the start of the with block, and the stop event is logged at the end of the with block. Each execution of a with block is considered a run. The step column is used to order the events within a run, for example to distinguish between multiple metrics logged in batches or loops within a run, such as is common in training neural networks.
The other columns logged are:
event: The type of event logged. This can be start, stop, metric, parameter, artifact, or system.
key: The key of the data logged. This can be any string, and is used to identify the data.
value: The value of the data logged. This can be any data type, and is used to store the data.
dt: The timestamp of the event. This is automatically logged by the experiment context manager.
node: The node on which the event was logged. This is automatically logged by the experiment context manager.
userid: The user id of the user who logged the event. This is automatically logged by the experiment context manager.
The data can be filtered by any of the columns, for example to retrieve only the metrics:
metrics = exp.data(event='metric')
| | experiment | run | step | event | key | value | dt | node | userid |
|----|--------------|-------|--------|:--------|:------|---------|----------------------------|:-------|:---------|
| 0 | iris | 1 | | metric | score | 0.5 | 2024-06-22 18:14:46.010676 | helio | patrick |
Experiments provide summary statistics, i.e. a summary of all the metrics logged:
exp.summary()
| | event | key | count | mean | std | min | 25% | 50% | 75% | max |
|----|---------|:--------|--------:|----------:|------------:|---------:|---------:|----------:|----------:|---------:|
| 0 | metric | score | 1 | 0.5 | nan | 0.5 | 0.5 | 0.5 | 0.5 | 0.5 |
| 1 | metric | latency | 6 | 0.0296315 | 0.0195344 | 0.013914 | 0.017405 | 0.0228145 | 0.0330427 | 0.066192 |
When we have recorded several runs of an experiment, we can compare the runs and plot metrics:
(exp
.data(event='metric', key='acc', run='*')
.plot(x='run', y='value'))
In addition to logging metrics, we can also log parameters, artifacts, and system statistics:
lr = LogisticRegression()
with om.runtime.experiment('iris') as exp:
exp.log_param('params', lr.get_params())
exp.log_artifact(lr, 'model')
exp.log_system()
exp.data()
| | experiment | run | step | event | key | value | dt | node | userid | name |
|----|--------------|-------|--------|:---------|:-------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------|:-------|:---------|:-------|
| 2 | iris | 6 | | start | start | | 2024-06-22 18:30:35.885176 | helio | patrick | nan |
| 1 | iris | 6 | | param | params | {'C': 1.0, 'class_weight': None, 'dual': False, 'fit_intercept': True, 'intercept_scaling': 1, 'l1_ratio': None, 'max_iter': 100, 'multi_class': 'auto', 'n_jobs': None, 'penalty': 'l2', 'random_state': None, 'solver': 'lbfgs', 'tol': 0.0001, 'verbose': 0, 'warm_start': False} | 2024-06-22 18:30:35.898604 | helio | patrick | nan |
| 0 | iris | 6 | | artifact | model | {'name': 'model', 'data': '.experiments/.artefacts/cc7d66e6686a40ba8b01baa73ee9ef3a', 'format': 'model'} | 2024-06-22 18:30:35.913042 | helio | patrick | model |
| 4 | iris | 6 | | system | system | {'platform': {'system': 'Linux', 'node': 'helio', 'release': '6.5.0-1023-oem', 'version': '#24-Ubuntu SMP PREEMPT_DYNAMIC Tue May 7 14:26:31 UTC 2024', 'machine': 'x86_64', 'processor': 'x86_64'}, 'python': 'CPython-3.10.14', 'packages': ['Babel==2.15.0', ...]} | 2024-06-22 18:30:35.913221 | helio | patrick | nan |
| 3 | iris | 6 | | stop | stop | | 2024-06-22 18:30:35.921282 | helio | patrick | nan |
Artifacts, as a result of exp.log_artifact() are stored as pickled objects, and can be retrieved as such:
exp.restore_artifacts()
[LogisticRegression()]
The system statistics are stored as a dictionary, and can be retrieved as such:
# exp.data() returns a dataframe
# -- the logged value is stored in the 'value' column
# -- the value column is a list of dictionaries
exp.data(event='system').value[0]
{'platform': {'system': 'Linux', 'node': 'helio', 'release': '6.5.0-1023-oem', 'version': '#24-Ubuntu SMP PREEMPT_DYNAMIC Tue May 7 14:26:31 UTC 2024', 'machine': 'x86_64', 'processor': 'x86_64'}, 'python': 'CPython-3.10.14', 'packages': ['Babel==2.15.0', ...]}
Further, we can log any extra data to the experiment, such as additional information:
with om.runtime.experiment('iris') as exp:
exp.log_extra(tag='v1.5')
exp.log_metric('score', .99)
exp.log_system()
exp.data()[['key', 'value', 'tag']]
| # | key | value | tag |
|----|--------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------|
| 1 | start | | nan |
| 0 | score | 0.99 | v1.5 |
| 3 | system | {'platform': {'system': 'Linux', ... } | v1.5 |
| 2 | stop | | v1.5 |