runArraySimulation() now correctly searches in .GlobalEnv for user
defined functions
manageWarnings(... suppress) argument now allows for partial matching
and other regex inputs
SimCollect() now automatically checks whether all files are expected to
be present via SimCheck()
runArraySimulation() gains a array2row function to allow array jobs
to index multiple conditions in the design object (default uses one
arrayID per row, the original behaviour)
runArraySimulation() gains parallel flag and friends to use multi-core
processing within array distributions. RNG numbers within the L'Ecuyer-CMRG
algorithm are incremented using parallel::nextRNGSubStream() within each
defined core
Better name checking when using the supported list inputs in runSimulation()
and runArraySimulation()
SimCollect() more efficient when combining a large number of files (e.g.,
greater than 5000 .rds files stored via runArraySimulation()). Gains a
dir argument for this purpose as well so that a full directory can be specified
SimCheck() repurposed to check for missing files for runArraySimulation()
Fix for SimCollect() when runArraySimulation() result contains
mixed warning outputs (reported by Michael Troung)
manageMessages() added in a similar spirit to manageWarnigns(), though
to change messages into either errors or warnings (default behavior is the
same as quiet())
manageWarnings() gains an suppress argument to specify explicit
warnings strings that can be suppressed (i.e., are known to be innocuous).
This provides better coding practice than the nuclear alternative
base::suppressWarnings()
convertWarnings() name changed to manageWarning() given its
increased functionality.
timeFormater() function added to isolate logic of SBATCH time specification
utility. Now used in several places of the package (e.g.,
runArraySimulation(), PBA(), SimSolve())
Switch to camel casing format in all functions (e.g.,
add_missing() -> addMissing(), gen_seeds() -> genSeeds(), etc). Exception
is that aggregate_simulations() has changed to SimCollect()
SimSolve() gains a predCI.tol argument to allow algorithm termination based
advertised precision of the estimates
runSimulation(..., control = list(store_Random.seeds)) logical added to
store all .Random.seed replication states. Generally not recommended due
to the size of these stored elements in larger simulations, however can be
useful for debugging purposes where errors or warnings are not thrown
runArraySimulation() added to better support distributing array's of jobs
on HPC clusters. Works best when combined with new expandDesign()
function (see next point) and the improved aggregate_simulations() behaviour
for more evenly distribution replication budgets across independent jobs.
An associated vignette file has been added to the package to provide
context and tutorial information for Slurm clusters
expandDesign() added to repeat the row conditions a number of
times instead of just once. This is useful when
exporting each condition independently to computing clusters, where each cluster
contains only a fraction of the target replications (see issue #33)
getArrayID() added to detect the array job ID (used
with runArraySimulation(..., arrayID))
aggregate_simulations() now requires explicit filename
argument to save the collapsed simulation information
aggregate_simulations() generalized to detect whether the Design conditions
have repeated row definitions and therefore should be conditionally averaged
over (see new expandDesign() function)
runArraySimulation() and runSimulation()'s control list gain
new max_time and max_RAM arguments to evaluate simulation
replications up until this time
or RAM storage constraint is reached. In the event that the target
replications are not reached the simulations up to this point, or the max
RAM storage has been reached, then on the partial results will be returned
(with a warning). This is mainly useful for HPC cluster jobs that require time
and RAM constraints (e.g., 4 days per job; 4GB of RAM),
where some jobs or simulation conditions may be more time/RAM consuming than
others (requested by Mikko Rönkkö)
Expose seed generation control per simulation condition via the
function gen_seeds(), which also automatically constructs proper
L'Ecuyer-CMRG seeds to be distributed across the runArraySimulation() jobs
SimSolve() function added to perform (stochastic) root-solving to estimate
specific criteria from simulation studies. Currently supports uni-root type
problems for continuous or discrete variables via the probabilistic
bisection algorithm with bolstering and interpolations (ProBABLI),
Brent's method, and the classical bisection approach, the latter two of which
can be problematic if the number of replications per iteration are too low
SFA() function added for fitting surrogate functional forms to
simulation results and subsequently solving specific roots. Supports single
root or multi-root applications, where by default the modelling is performed
via generalized linear models
runSimulation(..., store_results = TRUE) is now the default to automatically
store the results from Analyse() in the returned simulation object.
If RAM issues are suspected then save_results = TRUE is still
the recommended approach
convertWarnings() wrapper/post-hoc function added to convert specific
warning messages to errors during simulations. Useful when only a subset
of warnings are known to be problematic, while other warning messages
(whether known or not) are treated as provisionally innocuous
control gains a print_RAM logical argument to suppress printing the RAM
when verbose = TRUE. Disabling this can reduce execution time
as garbage collector (gc()) calls are avoided, which is required extract
the current RAM state. Setting verbose = FALSE will also automatically
disable the RAM and gc() calls and their overhead
Attach() now accepts matrix input objects, and gains a RStudio_flags
argument to generate syntax that suppresses false positives about variables
outside of the function's scope
Fix Github issue #26 related to extremely long warning/error messages
save_results_filename added to runSimulation() saving details to allow
asynchronous (though unchecked) file storage to the same results
directory (suggested by Jan Göttmann)
ECR() gain a complement logical to indicate whether parameter was
outside advertised interval (complement of coverage). Useful when CIs are
used as formal hypothesis tests (e.g., bootstrap CI tests for power)
runSimulation(..., extra_options) changed to control instead to
control less commonly used flags
createDesign() gains a fractional argument to support design
input structures from the FrF2 package for fractional factorial designs.
Useful when detecting main/low-dimensional interaction effects
across a large number of factor variables (suggested by Achim Zeileis).
Example added to the wiki to demonstrate its use
When summarise() function not supplied the Design input
is now appended to the results object when
using SimExtract(res, what = 'results'). Only supported when the results
object is a matrix-like structure
RAM element added to resulting objects to indicate the amount of RAM used
during each evaluation. This is particularly useful when using
runSimulation(..., store_results = TRUE) to inspect how much RAM is being
being consumed (otherwise, runSimulation(..., save_results = TRUE) should be
used if RAM storage is suspected to be an issue)
resummarise() and aggregate_simulation() now better support
the internally stored results terms when using store_results = TRUE
runSimulation(..., save = TRUE) changed to save = replications > 10
to only write temporary files when the replications are larger (less
hard-drive strain when initially testing simulation experiment with very
small replications)
hexsticker added to make SimDesign part of the cool-kids club
filename and save_results_dirname extractors added to SimExtract()
PBA() function added for probabilistic bisection algorithm, with associated
print() and plot() S3 methods
debug gains '-' structure to allow debugging on specific
rows of the design input. For instance, if the simulation
ran successfully until row 10, and unknown errors terminated the simulation,
then using runSimulation(..., debug = 'error-10')
will initiate the debugger on the first instance for the 10th row
conditions in the supplied design object
Progress reporting now includes abbreviated condition names and values in the console per condition
New function nc() to be used in situation where uniquely naming a vector or list
according to the object names is useful (cf. x <- c(A,B,C), which typically
returns an unnamed vector, to x <- nc(A,B,C), in which names(x) is "A" "B" "C").
This is mainly useful in the Analyse() step where objects must be named uniquely
in order to track the results in Summarise()
Added Bradley1978() for test of Bradley's (1978) robustness interval for
empirical detection/coverage rate statistics
runSimulation(..., Generate) can now be specified as a named list of functions
similar to Analyse(), however only the first valid data generation function
will be used as the constructor of the simulated data (see the new GenerateIf()
function to control the flow of these generation steps). This list input should
really only be used when the population generation functions are differ widely
depending on the condition under investigation
SimFunctions() adds a few new inputs for saving one or more files (save_structure),
defining one or more generate function (nGenerate), whether to include an extra
file for user-defined objects and functions (extra_file), and whether a basic
knitr::spin() header should be included when saving the files (spin_header)
Support the future package by using runSimulation(..., parallel = 'future')
to replace the built-in parallel processing inputs. Using the future package
approach makes several arguments to runSimulation() unnecessary as these
can be specified when defining future::plan() (e.g., cl, MPI, etc)
When using the future approach the progressr package is used.
Allows the progress bar to be started via progressr::with_progress() and
modified by the front-end user (see ?runSimulatino for an example
using progressr::handler())
extra_options gains support for .options.mpi to control the MPI properties
documented in doMPIquite() now removes the sunk connection temp file to save storage
issues (e.g., when distributing on Slurm)Attach() gains an omit argument to omit specific elements from being attached
to the working environment (default still attaches all objects supplied)Using a list definition for Analyse input now executes all functions by default
regardless of errors thrown. Error messages and seeds remain captured in the output,
however are labelled according to the number of errors that were observed (e.g.,
SimExtra(result, what = 'errors') may return column with
"ERROR: 2 INDEPENDENT ERRORS THROWN: ..."). Previous early termination default
can be reset by passing extra_options = list(try_all_analyse = FALSE) to runSimulation().
Special thanks to Mark Lai for bringing this to my attention on Issue #20
Added beep argument to runSimulation() to play a beep message via the beepr
Added RSE() function to compute the relative behaviour of the average standard error to the
standard deviation of a set of parameter estimates across the replications
(RSE = E(par_SEs) / SD(par_ests))
Bugfix for new list input for analysis functions when error raised (reported by Mark Lai)
SimExtract() gains a fuzzy argument to allow fuzzy matching of error and warning messages.
This helps collapse very similar errors messages in the recorded tables,
thereby improving how to discern any pattern in the errors/warnings (e.g., Messages such as
"ERROR: system is computationally singular: reciprocal condition number = 9.63735e-18" and
"ERROR: system is computationally singular: reciprocal condition number = 6.74615e-17" are
effectively the same, and so their number of recorded occurrences should be collapsed)
Added AnalyseIf() function to allow specific analysis function to be included explicitly. Useful
when the defined analysis function is not compatible with a row-condition in the Design object.
Only relevant when the analyse argument was defined as a named list of functions
The analyse argument to runSimulation() now accepts a named list of functions rather than a
single analysis function. This allows the user to separate the independent analyses into distinct
functional blocks rather than having all analyses within the same function, and potentially allows
for better modularity. The debug argument now also accepts the names of these respective
list elements to debug these function definitions quickly
SimFunctions() gains an nAnalyses argument to specify how many analysis functions should
be templated (default is 1, retaining the previous package defaults)
Various performance improvements to reduce execution overhead (e.g., REPLICATION ID now moved
to an extra_option as this was identified as a bottleneck)
Meta-statistical functions now support a fun(list, matrix) input form to compute element-wise
summaries that return a matrix structure
Summarise() can now return list arguments that can later be extracted
via SimExtract(sim, what = 'summarise'). Consequently, because list outputs are now
viable the purrr package has been added to the suggests list
Prevent aggregate_simulations() from overwriting files and directories accidentally. As well,
the auto-detection of suitable .rds files has been removed as explicitly stating the files/directories
to be aggregated is less error prone
Removed plyr::rbind.fill in favour of dplyr::bind_row(), which removed plyr as a dependency
Attach() now accepts multiple list-like objects as inputs
Added SimCheck() for checking the state of a long-running simulation via inspecting the main temp file
sessioninfo package used in placed of the traditional sessionInfo()
Print number of cores when parallel processing is in use
A number of arguments from runSimulation() moved into extra_options list argument to simplify documentation
Parallel processing now uses FORK instead of PSOCK when on Unix machines by default
More natural use of RPushbullet by changing the notification input into one that accepts a character vector ("none", "condition", "complete") to send pbPost() call. Also more informative in the default messages sent
Added "Empirical Supremum Rejection Sampling" method to rejectionSampling() to find better constant M (useful when there are local minimums in the f(x)/g(x) ratio)
rejectionSampling() made more general, with additional examples provided in the help files
Bootstrap CI estimates moved into runSimulation(), deprecating the less optimal SimBoot()
runSimulation(..., save=TRUE) now default to always store meta-information about the simulation state
Added renv to the suggests lists since it's useful to hard-store package versions used in simulations
data.frame objects largely replaced with tibble data frames instead as they render better for larger simulations
Support for rbind() and cbind() on final simulation results to add additional condition/meta-summary information
Use createDesign() instead of expand.grid() in code, which provides more structured information and flexibility
Added SimExtract() to extract important but silent information
Added stop_on_fatal logical argument to more aggressively terminate the simulation rather than do things more gracefully