1.1. Main idea and limitations of STOP

STOP was originally designed to be applied to Mixed-Integer Linear Programming solvers (common abbreviations are MILP, MIP, or IP). As of 2007 Aug 17 it has only been tested on such solvers.

We hope that STOP will also prove useful for other solvers. There are several assumptions that underlie STOP's method:

• parameters take discrete values

• for a particular class of instances, efficiency depends primarily on just a few of these parameters, though which parameters those are may be unknown and may differ from one class of instances to another

• the number of parameter values for each parameter is small, though the number of parameters may be relatively large

These assumptions come from the fact that STOP is combinatorially in nature. It tests many parameter settings, carefully chosen to cover many combinations of parameter values. This is the reason that parameters must take discrete values. Of course, it may be possible to discretize continuous parameters.

Our observation for MILP solvers is that for particular classes of instances, it is usually the case that just a few (two or three) of the parameters will be most important. Presumably this has to do with the underlying structure of the class. Therefore, it is not necessary to find perfect values for every parameter: it is enough to find a setting that has good values for these key parameters. The good news is that you do not need to know which parameters will be the key ones for your class; if you knew that, you probably wouldn't need STOP!

If your program will only work well when all parameters have good values, then STOP will probably not be useful.

The assumption that the number of parameter value is small is because STOP is more sensitive to the maximum number of parameter value than it is to the number of parameters. See the section on creating a new interface for more explanation on why this is the case.

1.2. Possible uses for STOP

STOP requires a significant amount of computer time (though little of the user's time), since it needs to try many settings. With this in mind, it is not always appropriate to use STOP. In particular, if you have a new instance and just want to know the solution quickly, STOP is probably not helpful. (With some modifications there could be a use of STOP for very time-consuming instances, but not in its current form.)

We imagine several possible uses for STOP:

• A user wishes to solve many similar instances (at least dozens), so an up-front time investment to get good parameters may pay off.

• A user wishes to solve some instances when time is critical, so a prior investment of "off-line" computer time may be worthwhile, even though there is not a net time savings.

• A user is doing an experiment to compare algorithms or versions of algorithms and wants to compare each with its own good parameters. A manual choice of parameters is likely to be subject to criticism, while STOP may allow a more fair comparison of each algorithm at its best.

1.3. The name

The name STOP was chosen because it's humorous for us:

• I wish this solver would STOP!

• STOP worrying about finding good parameter settings

We designed it with optimization tools in mind, which is why 'optimization' is part of the name. It could be that it will be useful for other types of software as well. If so, perhaps we will have to consider changing the name.

2.1. What is an interface

A STOP interface contains the code necessary for STOP to be used with a particular solver, with a particular metric, and with a particular set of parameters and parameter values. Using another solver or metric requires a different interface. Using the same solver but different parameters or even just different parameter values also requires a different interface (though it should be easy to modify the existing interface for such changes).

Several interfaces are available with the STOP distribution. More may be added in the future. If you need an interface that is not provided, then see the section on creating a new interface.

2.2. Installation

To install STOP, there are four main steps:

1. Download the files

2. Configure the Makefile

3. Edit STOP.cpp to select an interface

4. Compile and link

Downloading the files should be easy. You can copy them to any convenient location.

Configuring the Makefile is the hardest part. For most interfaces, STOP needs to be linked directly to the solver using the solver's API. Before compiling, you need to let STOP know the location of the solver's header files and library files on your machine. This is done with the Makefile. See the instructions in the Makefile for telling STOP where to find the appropriate files. Note that you only need to specify details for the interface you plan to use.

Next it is necessary to select an interface. At present this is done with an #include statement in the file STOP.cpp. Near the beginning of the file, look for the #include statements for interfaces. All of them should be commented out except for one. Edit if necessary so that the desired interface is the only one that is included.

Compiling and linking should be easy if your system has the compiler expected by the Makefile. For unix-like system, you should be fine with GNU Make and GCC. In this case, the command make STOP_xxx should do it, where xxx is changed to match your interface's target in the Makefile.

For Windows, there is a project file for MS Visual C++. Instead of editing the Makefile, you will need to change settings in MS Visual C++ so that it can find the solver's include files and libraries.

If compiling and linking was successful, then STOP is ready to run.

2.3. Using STOP

At run time, the following details must be specified:

• a file containing a list of instance names

• the number of settings to try

• the selection method (pairwise coverage, greedy conflict heuristic, or random)

• machine learning method (none, regression tree, or neural network)

It's also possible to specify some optional settings. Perhaps most useful is a time limit on solver runs, given with the option --time-limit.

Usage details are provided with the command STOP --help.

Some examples of commands are given below. Note that the default selection method is the greedy conflict heuristic, while the default machine learning is none.

• STOP -n 32 instance_file: tests 32 settings chosen with the default selection method (conflict heuristic) and no machine learning

• STOP -n 32 --time-limit 600 instance_file: same as previous, but there is a time limit of 600 seconds on individual solver runs

• STOP -n 32 --time-limit 600 --pairwise instance_file: same as previous, but use pairwise coverage for selection (if possible)

• STOP -n 32 --time-limit 600 --pairwise --regression-tree -a 16 instance_file: same as previous, but uses a regression tree to guide the selection of 16 additional settings (48 total settings)

• STOP -n 32 --time-limit 600 --pairwise --neural-net -a 16 instance_file: same as previous, but uses an artificial neural network to guide the selection of 16 additional settings (48 total settings)

3.1. The structure of an interface file

For any changes to an interface file, it is helpful to understand the structure of the code. The interface file defines the class STOP_interface. If you're not familiar with object-oriented programming, you can think of this as a data type with associated functions.

In most cases, there are only seven functions that may need to be changed:

• STOP_interface (constructor): This function is called once when STOP starts. It defines the parameters and parameter values that will be used and does any one-time initialization that the solver may require.

• ~STOP_interface (destructor): This function is called once when STOP is finished. It does any one-time clean-up. In many cases, nothing is necessary here.

• initialize_metric: This function is called once for each setting considered. Whether or not anything is necessary depends on the metric.

• get_metric: This function returns the metric for a particular setting after all instances have been solved.

• set_parameters: This function is called once for each setting considered.

• load_instance: This function is called once for each instance that is solved.

• execute_program: This function is called to actually solve one instance.

Simplified pseudocode for STOP to illustrate the role of these seven function is given below:

1. Call STOP_interface constructor

2. Determine initial settings to test

3. For each setting:

   1. Call initialize_metric

   2. Call set_parameters

   3. For each instance:

      1. Call load_instance

      2. Call execute_program

   4. Call get_metric

4. If machine learning is being used, then select additional settings to test and test them using the same process above

5. Call ~STOP_interface destructor

3.2. Changing parameters or parameter values

We first consider the easiest case. We assume that there is already an interface for your solver with the correct metric, but you want to change the parameters or parameter values that STOP will consider.

To do this, copy the interface file you wish to modify, which will have a name like interface_XXX.hpp. Give it any new name you like. In STOP.cpp, add an #include statement to include the new interface file and comment out any other interface files.

In the interface file, parameters are specified in the STOP_interface constructor. These commands look like par_name.push_back("Some label"); Each parameter that is considered should be added to the par_name vector and the number of parameter values it has should be added to the par_levels vector. The description for the parameters can be whatever you like; it would be good to choose something that will be clear to the user.

Immediately after that in the interface file (still in STOP_interface) is where the parameter values are specified with commands of the form par_level[2].push_back(). The numerical values that STOP uses internally will start at 0 and increase, using the order that the parameter values are specified here. Any descriptive strings can be used here.

Note that the interface still doesn't know how to actually change the parameters. Depending on the interface, this could be specified in the set_parameters method, the load_instance method, or the execute_program method. It is the programmer's responsibility to make sure that the order given for the parameter values earlier is matched up correctly to the solver's internal parameters.

3.3. Changing the metric for comparison

The interface file may be used to specify an arbitrary metric for comparing settings. For example, this could be the sum of the solution times (to proven optimality) for all instances, or it could be the average proven gap after some fixed amount of time.

The metric is usually most affected by the execute_program function. After the solver is run, any information for the metric should be stored. Most solvers have a double metric variable, but it's possible to use any set of variables you need, as long as the final value is returned as a double.

Make sure that get_metric and initialize_metric are set correctly for the metric in question.

3.4. Creating a new interface

Writing a new interface requires a higher level of knowledge and commitment than changing an existing interface. By looking at existing interfaces you can get a feel for how they have typically been done and can probably find an existing interface that is closest to yours. It will probably be easiest to copy that interface and then modify it.

Use the information about the key functions given at the start of this chapter to decide how your interface will work best. Remember that you can freely add variables to the STOP_interface class if they will make things easier.

Remember to edit STOP.cpp to include your new interface for testing. It's best to test on very easy instances so you can quickly see the flow of STOP itself.

It's best to check that each solver run was successful and include an error message if it was not successful. Not all existing interfaces do this equally well, but ideally every one of them would check carefully whether the solver was successful.