R Hartmann6 with Constraints¶
This notebook is an example of a single objective R model optimization using a Hartmann6 function utilizing parameter constraints as well as an additional simulated black box constraint. Here our parameter constraints specify that some linear combination of some of the parameters needs to be under some constant (i.e. a budget constraint). Our simulated black box constraint is any function that we could theoretically calculate in our run_model.R script as an additional constraint (in this case we just use the l2 norm of the parameters).
The Hartmann6 function is a synthetic function that is often used as a stand in for a model that is expensive to evaluate. The function has 6 dimensions and is defined on the unit hypercube.
See the folder r_hartmann for the R script and the configuration file.
1# This notebook uses hidden cells to import and display things so
2# that it can be ran regullary to make sure the documentation
3# is up to date and not broken (as opposed to a markdown file
4# that would have an exmaple written once and get out of date
5# as the code base changes)
6
7# These hidden cells are only responsible for rerunning
8# the documentation to ensure it is correct.
9# the actual relavent part of the documentation
10# are the non hidden parts
Configuration File¶
config_constraints.yaml
objective:
# This is the metric(s) we are trying to optimize
# This exact number will be used in our run_model.R script later
metrics:
- name: Hartmann6
# This is any additional constraint you want to optionally add to the optimization
# This often might be a black box constraint that you want to enforce
# Or a budget constraint that you want to enforce
outcome_constraints:
- l2norm <= 1.25
parameters:
# Parameters can be 3 types: range, fixed, or choice
# with fixed being inferred if it is a single value (i.e. x1: 3)
# Parameters can be of type int, float, str, or bool
# and are inferred if not specified
x1:
type: range
bounds: [0.0, 1.0]
x2:
type: range
bounds: [0.0, 1.0]
x3:
type: range
bounds: [0.0, 1.0]
x4:
type: range
bounds: [0.0, 1.0]
x5:
type: range
bounds: [0.0, 1.0]
x6:
type: range
bounds: [0.0, 1.0]
parameter_constraints:
# Optional, list of constraints on the parameter space, these are linear constraints
- .5*x1 + x2 <= 1.5
n_trials: 10
script_options:
# This tells BOA what script from any language to run to evaluate the model
run_model: Rscript run_model_w_l2norm.R
Run Model Wrapper Script¶
run_model.R
# load in any libraries and modules we need
library(jsonlite)
source("./hartmann6.R")
trial_dir <- commandArgs(trailingOnly=TRUE)[1]
data <- read_json(path=file.path(trial_dir, "parameters.json"))
res <- hartmann6(unlist(data))
write(toJSON(list(Hartmann6=res, l2norm=sum(unlist(data)**2)**(1/2))), file.path(trial_dir, "output.json"))
We also use a function called hartman6 which is a 6 dimensional version of the synthetic hartman function as the stand in for our model function. The code is below. You would substitute this for any call your model, be it local call to your own R model, a system call to a fortran model wrapped in your R script, or perhaps a some code that launches an HPC job and collects the results.
hartmann6.R
1Code(hartmann6_script)
hartmann6 <- function(X) {
out <- tryCatch(
{
alpha <- c(1.0, 1.2, 3.0, 3.2)
A <- c(10, 3, 17, 3.5, 1.7, 8,
0.05, 10, 17, 0.1, 8, 14,
3, 3.5, 1.7, 10, 17, 8,
17, 8, 0.05, 10, 0.1, 14)
A <- matrix(A, 4, 6, byrow=TRUE)
P <- 10^(-4) * c(1312, 1696, 5569, 124, 8283, 5886,
2329, 4135, 8307, 3736, 1004, 9991,
2348, 1451, 3522, 2883, 3047, 6650,
4047, 8828, 8732, 5743, 1091, 381)
P <- matrix(P, 4, 6, byrow=TRUE)
Xmat <- matrix(rep(X,times=4), 4, 6, byrow=TRUE)
inner_sum <- rowSums(A[,1:6]*(Xmat-P[,1:6])^2)
outer_sum <- sum(alpha * exp(-inner_sum))
y <- -outer_sum
return(y)
},
error=function(cond) {
return(NA)
}
)
return(out)
}
Running our script¶
To run our script we just need to pass the config file to BOA’s CLI
python -m boa --config-file path/to/config.yaml
or
python -m boa -c path/to/config.yaml
[WARNING 07-19 00:00:36] ax.service.utils.with_db_settings_base: Ax currently requires a sqlalchemy version below 2.0. This will be addressed in a future release. Disabling SQL storage in Ax for now, if you would like to use SQL storage please install Ax with mysql extras via `pip install ax-platform[mysql]`.
[INFO 07-19 00:00:37] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter x1. If that is not the expected value type, you can explicitly specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 07-19 00:00:37] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter x2. If that is not the expected value type, you can explicitly specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 07-19 00:00:37] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter x3. If that is not the expected value type, you can explicitly specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 07-19 00:00:37] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter x4. If that is not the expected value type, you can explicitly specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 07-19 00:00:37] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter x5. If that is not the expected value type, you can explicitly specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 07-19 00:00:37] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter x6. If that is not the expected value type, you can explicitly specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 07-19 00:00:37] ax.service.utils.instantiation: Created search space: SearchSpace(parameters=[RangeParameter(name='x1', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x2', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x3', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x4', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x5', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x6', parameter_type=FLOAT, range=[0.0, 1.0])], parameter_constraints=[ParameterConstraint(0.5*x1 + 1.0*x2 <= 1.5)]).
[INFO 07-19 00:00:37] ax.modelbridge.dispatch_utils: Using Models.BOTORCH_MODULAR since there are more ordered parameters than there are categories for the unordered categorical parameters.
[INFO 07-19 00:00:37] ax.modelbridge.dispatch_utils: Calculating the number of remaining initialization trials based on num_initialization_trials=None max_initialization_trials=None num_tunable_parameters=6 num_trials=None use_batch_trials=False
[INFO 07-19 00:00:37] ax.modelbridge.dispatch_utils: calculated num_initialization_trials=12
[INFO 07-19 00:00:37] ax.modelbridge.dispatch_utils: num_completed_initialization_trials=0 num_remaining_initialization_trials=12
[INFO 07-19 00:00:37] ax.modelbridge.dispatch_utils: `verbose`, `disable_progbar`, and `jit_compile` are not yet supported when using `choose_generation_strategy` with ModularBoTorchModel, dropping these arguments.
[INFO 07-19 00:00:37] ax.modelbridge.dispatch_utils: Using Bayesian Optimization generation strategy: GenerationStrategy(name='Sobol+BoTorch', steps=[Sobol for 12 trials, BoTorch for subsequent trials]). Iterations after 12 will take longer to generate due to model-fitting.
[INFO 07-19 00:00:37] Scheduler: `Scheduler` requires experiment to have immutable search space and optimization config. Setting property immutable_search_space_and_opt_config to `True` on experiment.
[INFO 2024-07-19 00:00:37,576 MainProcess MainThread {controller.py:165}] boa:
##############################################
BOA Experiment Run
Output Experiment Dir: [/path/to/your/dir/]/boa_runs_20240719T000037
Scheduler File Path: [/path/to/your/dir/]/boa_runs_20240719T000037/scheduler.json
Optimization CSV File Path: [/path/to/your/dir/]/boa_runs_20240719T000037/optimization.csv
Start Time: 20240719T000037
Version: 0.10.3
##############################################
[INFO 07-19 00:00:37] Scheduler: Running trials [0]...
[INFO 2024-07-19 00:00:37,588 MainProcess ThreadPoolExecutor-0_0 {script_wrapper.py:322}] boa: Rscript run_model_w_l2norm.R
[INFO 2024-07-19 00:00:37,589 MainProcess ThreadPoolExecutor-0_0 {script_wrapper.py:328}] boa: {'config_path': PosixPath('[/path/to/your/dir/]/config_constraints.yaml'), 'config_dir': PosixPath('[/path/to/your/dir/]'), 'config_dir_name': 'r_hartmann', 'config_file_name': 'config_constraints.yaml', 'x1': 0.07178448885679245, 'x2': 0.8356549143791199, 'x3': 0.0035144714638590813, 'x4': 0.7863138318061829, 'x5': 0.07032205909490585, 'x6': 0.5706493258476257, 'trial_dir': PosixPath('[/path/to/your/dir/]/boa_runs_20240719T000037/000000')}
[INFO 2024-07-19 00:00:37,590 MainProcess ThreadPoolExecutor-0_0 {script_wrapper.py:330}] boa: Rscript run_model_w_l2norm.R
[INFO 07-19 00:00:38] Scheduler: Running trials [1]...
[INFO 2024-07-19 00:00:38,604 MainProcess ThreadPoolExecutor-1_0 {script_wrapper.py:322}] boa: Rscript run_model_w_l2norm.R
[INFO 2024-07-19 00:00:38,605 MainProcess ThreadPoolExecutor-1_0 {script_wrapper.py:328}] boa: {'config_path': PosixPath('[/path/to/your/dir/]/config_constraints.yaml'), 'config_dir': PosixPath('[/path/to/your/dir/]'), 'config_dir_name': 'r_hartmann', 'config_file_name': 'config_constraints.yaml', 'x1': 0.3690582951530814, 'x2': 0.8248181883245707, 'x3': 0.1528281858190894, 'x4': 0.6253101080656052, 'x5': 0.8966255439445376, 'x6': 0.08540637511759996, 'trial_dir': PosixPath('[/path/to/your/dir/]/boa_runs_20240719T000037/000001')}
[INFO 2024-07-19 00:00:38,606 MainProcess ThreadPoolExecutor-1_0 {script_wrapper.py:330}] boa: Rscript run_model_w_l2norm.R
[INFO 07-19 00:00:39] Scheduler: Running trials [2]...
[INFO 2024-07-19 00:00:39,622 MainProcess ThreadPoolExecutor-2_0 {script_wrapper.py:322}] boa: Rscript run_model_w_l2norm.R
[INFO 2024-07-19 00:00:39,623 MainProcess ThreadPoolExecutor-2_0 {script_wrapper.py:328}] boa: {'config_path': PosixPath('[/path/to/your/dir/]/config_constraints.yaml'), 'config_dir': PosixPath('[/path/to/your/dir/]'), 'config_dir_name': 'r_hartmann', 'config_file_name': 'config_constraints.yaml', 'x1': 0.7129219304770231, 'x2': 0.449141058139503, 'x3': 0.5846446640789509, 'x4': 0.1756510604172945, 'x5': 0.13977822195738554, 'x6': 0.8030654825270176, 'trial_dir': PosixPath('[/path/to/your/dir/]/boa_runs_20240719T000037/000002')}
[INFO 2024-07-19 00:00:39,624 MainProcess ThreadPoolExecutor-2_0 {script_wrapper.py:330}] boa: Rscript run_model_w_l2norm.R
[INFO 07-19 00:00:40] Scheduler: Running trials [3]...
[INFO 2024-07-19 00:00:40,640 MainProcess ThreadPoolExecutor-3_0 {script_wrapper.py:322}] boa: Rscript run_model_w_l2norm.R
[INFO 2024-07-19 00:00:40,641 MainProcess ThreadPoolExecutor-3_0 {script_wrapper.py:328}] boa: {'config_path': PosixPath('[/path/to/your/dir/]/config_constraints.yaml'), 'config_dir': PosixPath('[/path/to/your/dir/]'), 'config_dir_name': 'r_hartmann', 'config_file_name': 'config_constraints.yaml', 'x1': 0.2241874411702156, 'x2': 0.3227615049108863, 'x3': 0.5689222635701299, 'x4': 0.9998589232563972, 'x5': 0.9343658247962594, 'x6': 0.16477635223418474, 'trial_dir': PosixPath('[/path/to/your/dir/]/boa_runs_20240719T000037/000003')}
[INFO 2024-07-19 00:00:40,642 MainProcess ThreadPoolExecutor-3_0 {script_wrapper.py:330}] boa: Rscript run_model_w_l2norm.R
[INFO 07-19 00:00:40] Scheduler: Running trials [4]...
[INFO 2024-07-19 00:00:40,664 MainProcess ThreadPoolExecutor-4_0 {script_wrapper.py:322}] boa: Rscript run_model_w_l2norm.R
[INFO 2024-07-19 00:00:40,666 MainProcess ThreadPoolExecutor-4_0 {script_wrapper.py:328}] boa: {'config_path': PosixPath('[/path/to/your/dir/]/config_constraints.yaml'), 'config_dir': PosixPath('[/path/to/your/dir/]'), 'config_dir_name': 'r_hartmann', 'config_file_name': 'config_constraints.yaml', 'x1': 0.12201976124197245, 'x2': 0.5560232866555452, 'x3': 0.4776177816092968, 'x4': 0.5704949209466577, 'x5': 0.7563984282314777, 'x6': 0.6445742156356573, 'trial_dir': PosixPath('[/path/to/your/dir/]/boa_runs_20240719T000037/000004')}
[INFO 2024-07-19 00:00:40,668 MainProcess ThreadPoolExecutor-4_0 {script_wrapper.py:330}] boa: Rscript run_model_w_l2norm.R
[INFO 07-19 00:00:41] Scheduler: Running trials [5]...
[INFO 2024-07-19 00:00:41,689 MainProcess ThreadPoolExecutor-5_0 {script_wrapper.py:322}] boa: Rscript run_model_w_l2norm.R
[INFO 2024-07-19 00:00:41,691 MainProcess ThreadPoolExecutor-5_0 {script_wrapper.py:328}] boa: {'config_path': PosixPath('[/path/to/your/dir/]/config_constraints.yaml'), 'config_dir': PosixPath('[/path/to/your/dir/]'), 'config_dir_name': 'r_hartmann', 'config_file_name': 'config_constraints.yaml', 'x1': 0.5878723105415702, 'x2': 0.5524903871119022, 'x3': 0.8375883558765054, 'x4': 0.8262278363108635, 'x5': 0.5279082870110869, 'x6': 0.45386915002018213, 'trial_dir': PosixPath('[/path/to/your/dir/]/boa_runs_20240719T000037/000005')}
[INFO 2024-07-19 00:00:41,692 MainProcess ThreadPoolExecutor-5_0 {script_wrapper.py:330}] boa: Rscript run_model_w_l2norm.R
[INFO 07-19 00:00:42] Scheduler: Running trials [6]...
[INFO 2024-07-19 00:00:42,710 MainProcess ThreadPoolExecutor-6_0 {script_wrapper.py:322}] boa: Rscript run_model_w_l2norm.R
[INFO 2024-07-19 00:00:42,712 MainProcess ThreadPoolExecutor-6_0 {script_wrapper.py:328}] boa: {'config_path': PosixPath('[/path/to/your/dir/]/config_constraints.yaml'), 'config_dir': PosixPath('[/path/to/your/dir/]'), 'config_dir_name': 'r_hartmann', 'config_file_name': 'config_constraints.yaml', 'x1': 0.2713579358533025, 'x2': 0.5852999957278371, 'x3': 0.9166425503790379, 'x4': 0.6598202046006918, 'x5': 0.6381644681096077, 'x6': 0.9400179041549563, 'trial_dir': PosixPath('[/path/to/your/dir/]/boa_runs_20240719T000037/000006')}
[INFO 2024-07-19 00:00:42,714 MainProcess ThreadPoolExecutor-6_0 {script_wrapper.py:330}] boa: Rscript run_model_w_l2norm.R
[INFO 07-19 00:00:43] Scheduler: Running trials [7]...
[INFO 2024-07-19 00:00:43,733 MainProcess ThreadPoolExecutor-7_0 {script_wrapper.py:322}] boa: Rscript run_model_w_l2norm.R
[INFO 2024-07-19 00:00:43,735 MainProcess ThreadPoolExecutor-7_0 {script_wrapper.py:328}] boa: {'config_path': PosixPath('[/path/to/your/dir/]/config_constraints.yaml'), 'config_dir': PosixPath('[/path/to/your/dir/]'), 'config_dir_name': 'r_hartmann', 'config_file_name': 'config_constraints.yaml', 'x1': 0.7992269936949015, 'x2': 0.5686128176748753, 'x3': 0.02279709931463003, 'x4': 0.6886428622528911, 'x5': 0.6163399536162615, 'x6': 0.5422101132571697, 'trial_dir': PosixPath('[/path/to/your/dir/]/boa_runs_20240719T000037/000007')}
[INFO 2024-07-19 00:00:43,737 MainProcess ThreadPoolExecutor-7_0 {script_wrapper.py:330}] boa: Rscript run_model_w_l2norm.R
[INFO 07-19 00:00:44] Scheduler: Running trials [8]...
[INFO 2024-07-19 00:00:44,757 MainProcess ThreadPoolExecutor-8_0 {script_wrapper.py:322}] boa: Rscript run_model_w_l2norm.R
[INFO 2024-07-19 00:00:44,759 MainProcess ThreadPoolExecutor-8_0 {script_wrapper.py:328}] boa: {'config_path': PosixPath('[/path/to/your/dir/]/config_constraints.yaml'), 'config_dir': PosixPath('[/path/to/your/dir/]'), 'config_dir_name': 'r_hartmann', 'config_file_name': 'config_constraints.yaml', 'x1': 0.16274631768465042, 'x2': 0.790202328003943, 'x3': 0.11078974977135658, 'x4': 0.19785477686673403, 'x5': 0.28264366928488016, 'x6': 0.10864654462784529, 'trial_dir': PosixPath('[/path/to/your/dir/]/boa_runs_20240719T000037/000008')}
[INFO 2024-07-19 00:00:44,761 MainProcess ThreadPoolExecutor-8_0 {script_wrapper.py:330}] boa: Rscript run_model_w_l2norm.R
[INFO 07-19 00:00:45] Scheduler: Running trials [9]...
[INFO 2024-07-19 00:00:45,782 MainProcess ThreadPoolExecutor-9_0 {script_wrapper.py:322}] boa: Rscript run_model_w_l2norm.R
[INFO 2024-07-19 00:00:45,784 MainProcess ThreadPoolExecutor-9_0 {script_wrapper.py:328}] boa: {'config_path': PosixPath('[/path/to/your/dir/]/config_constraints.yaml'), 'config_dir': PosixPath('[/path/to/your/dir/]'), 'config_dir_name': 'r_hartmann', 'config_file_name': 'config_constraints.yaml', 'x1': 0.719412000849843, 'x2': 0.7022498846054077, 'x3': 0.16587317269295454, 'x4': 0.5902803745120764, 'x5': 0.46134545747190714, 'x6': 0.9697931436821818, 'trial_dir': PosixPath('[/path/to/your/dir/]/boa_runs_20240719T000037/000009')}
[INFO 2024-07-19 00:00:45,786 MainProcess ThreadPoolExecutor-9_0 {script_wrapper.py:330}] boa: Rscript run_model_w_l2norm.R
[INFO 07-19 00:00:46] Scheduler: Retrieved COMPLETED trials: 0 - 9.
[INFO 07-19 00:00:46] Scheduler: Fetching data for trials: 0 - 9.
[WARNING 07-19 00:00:46] ax.service.utils.report_utils: Column reason missing for all trials. Not appending column.
[INFO 2024-07-19 00:00:46,928 MainProcess MainThread {storage.py:303}] boa: Saved optimization parametrization and objective to `[/path/to/your/dir/]/boa_runs_20240719T000037/optimization.csv`.
[INFO 2024-07-19 00:00:46,955 MainProcess MainThread {storage.py:63}] boa: Saved JSON-serialized state of optimization to `[/path/to/your/dir/]/boa_runs_20240719T000037/scheduler.json`.
Boa version: 0.10.3
[INFO 2024-07-19 00:00:46,971 MainProcess MainThread {scheduler.py:86}] boa: Trials so far: 10
Currently running trials: []
Will Produce next trials from generation step: Sobol
Best trial so far: {8: {'Hartmann6': -0.2906, 'l2norm': 0.8911}}
[WARNING 07-19 00:00:46] ax.service.utils.report_utils: Column reason missing for all trials. Not appending column.
[INFO 2024-07-19 00:00:46,991 MainProcess MainThread {storage.py:303}] boa: Saved optimization parametrization and objective to `[/path/to/your/dir/]/boa_runs_20240719T000037/optimization.csv`.
[INFO 2024-07-19 00:00:47,016 MainProcess MainThread {storage.py:63}] boa: Saved JSON-serialized state of optimization to `[/path/to/your/dir/]/boa_runs_20240719T000037/scheduler.json`.
Boa version: 0.10.3
[INFO 2024-07-19 00:00:47,032 MainProcess MainThread {scheduler.py:86}] boa: Trials so far: 10
Currently running trials: []
Will Produce next trials from generation step: Sobol
Best trial so far: {8: {'Hartmann6': -0.2906, 'l2norm': 0.8911}}
[WARNING 07-19 00:00:47] ax.service.utils.report_utils: Column reason missing for all trials. Not appending column.
[INFO 2024-07-19 00:00:47,063 MainProcess MainThread {controller.py:188}] boa:
##############################################
Trials Completed!
BOA Experiment Run
Output Experiment Dir: [/path/to/your/dir/]/boa_runs_20240719T000037
Scheduler File Path: [/path/to/your/dir/]/boa_runs_20240719T000037/scheduler.json
Optimization CSV File Path: [/path/to/your/dir/]/boa_runs_20240719T000037/optimization.csv
Start Time: 20240719T000037
Version: 0.10.3
End Time: 20240719T000047
Total Run Time: 9.457314491271973
trial_index arm_name trial_status ... x4 x5 x6
0 0 0_0 COMPLETED ... 0.786314 0.070322 0.570649
1 1 1_0 COMPLETED ... 0.625310 0.896626 0.085406
2 2 2_0 COMPLETED ... 0.175651 0.139778 0.803065
3 3 3_0 COMPLETED ... 0.999859 0.934366 0.164776
4 4 4_0 COMPLETED ... 0.570495 0.756398 0.644574
5 5 5_0 COMPLETED ... 0.826228 0.527908 0.453869
6 6 6_0 COMPLETED ... 0.659820 0.638164 0.940018
7 7 7_0 COMPLETED ... 0.688643 0.616340 0.542210
8 8 8_0 COMPLETED ... 0.197855 0.282644 0.108647
9 9 9_0 COMPLETED ... 0.590280 0.461345 0.969793
[10 rows x 13 columns]
##############################################