How to do IO args for parallel cmdstan?

Summary:

I think stan-dev/stan#3033 should hopefully go in next week or so and I'd like to figure out some of the command line arguments and processing those input args related to running multiple chains in a Stan program

We need to figure out

The actual arg to specify multiple parallel chains
How to handle args of initial values of parameters, inverse mass matrices, and output files

Docs also need updated to describe the behavior of the rng with multiple chains as well as any arguments

New args

There's a few ways to specify the number of chains. What I have in the prototype is

STAN_NUM_THREADS=20 examples/dawid/dawid sample num_samples=500 \
  num_warmup=500 data file=examples/dawid/caries_dump.R chains n_chains=8

But idt I like this because, what other args would go into chains? I think what might be better here is to have a parallel "meta arg" so users would write

STAN_NUM_THREADS=20 examples/dawid/dawid sample num_samples=500 \
  num_warmup=500 data file=examples/dawid/caries_dump.R parallel n_chains=8

We want to add this meta arg for specifying the number of parameters and gpu info anyway so I think this is a good opportunity to add it so users can write things like

examples/dawid/dawid sample num_samples=500 \
  num_warmup=500 data file=examples/dawid/caries_dump.R parallel n_chains=8 n_threads=20

Does anyone disagree with this?

Multiple IO

How to handle multiple init and output files is kind of tricky. The definition for init right now is

init=<string>
  Initialization method: "x" initializes randomly between [-x, x], "0" initializes to 0, anything else 
    identifies a file of values
  Valid values: All
  Defaults to "2"

So the only issue is the file of values. I think what we could do here is a simple rule like

init=<string>
  Initialization method: "x" initializes randomly between [-x, x], "0" initializes to 0, anything else
   identifies a file of values. If `n_chain` is greater than one the program will first search
   for "path/to/{init_filename}_{1:n_chain}.{file_ending}" and if not found will then search for
   a file "path/to/{init_filename}.{file_ending}"
  Valid values: All
  Defaults to "2"

So if the user specifies their init file as "my_init.json" we will first look for "my_init_1.json", "my_init_2.json", etc. and if that fails then we look for "my_init.json".

I think we can do the same thing for the initial mass matrix. For the output files we just take the output file they specify and make a new set out output files with a _1:n_chain before the file extension. If anyone has better ideas I'm all ears!

Current Version:

v2.26.1

one way to figure this out is to write the docs first - here's the chapter in the CmdStan manual that goes over the set of arguments and the (expletive deleted) argument hierarchy: https://mc-stan.org/docs/2_26/cmdstan-guide/command-line-interface-overview.html

this is NUTS-HMC specific, correct? I'm happy to help with the CmdStan argument parser wrangling.

regarding proposal for naming of input files, this seems like a reasonable hack for the current CmdStan arg parser.

we also need to consider what happens for all of the outputs - this includes Stan CSV output files, diagnostics file, profile file.

note that the CmdStan process has a single stdout and stderr and progress messages ("refresh") are written to stdout. will this work as expected for parallel cmdstan?

one way to figure this out is to write the docs first - here's the chapter in the CmdStan manual that goes over the set of arguments and the (expletive deleted) argument hierarchy: https://mc-stan.org/docs/2_26/cmdstan-guide/command-line-interface-overview.html

Could be good! I'll make a PR.

this is NUTS-HMC specific, correct? I'm happy to help with the CmdStan argument parser wrangling.

For now yes, but the scheme can be applied to all of our samplers / optimizers

we also need to consider what happens for all of the outputs - this includes Stan CSV output files, diagnostics file, profile file.

Oh good point. I think those should be per chain so I think my scheme for output should work

note that the CmdStan process has a single stdout and stderr and progress messages ("refresh") are written to stdout. will this work as expected for parallel cmdstan?

So std::cout and friends are by the standard threadsafe, but what can happen is that multiple threads write to the streams buffer one after another which can cause the output to be all jarbled and goofy looking. Like for example here

  void operator()(const std::string& message) {
    *output_ << comment_prefix_ << message << std::endl;
  }

Those multiple << are each functions which on separate threads can happen in an interleaved manner leaving a weird looking output.

I think a simple solution is in the stream writer to do something like in this stackoverflow post

void operator()(const std::string& message) {
  std::stringstream streamer; 
  streamer << comment_prefix_ << message << std::endl;
  *output_ << streamer;
}

By writing to a stringstream first those writes happen within the thread and then there's no interweaving when writing to std::cout etc. since there's only one << buffer write for each thread

I think my plan is to work out the prototype a bit more to make sure all the stuff I'm saying about I/O makes sense, then update the docs with those things

I think a simple solution is in the stream writer to do something like in this stackoverflow post

void operator()(const std::string& message) {
  std::stringstream streamer; 
  streamer << comment_prefix_ << message << std::endl;
  *output_ << streamer;
}

By writing to a stringstream first those writes happen within the thread and then there's no interweaving when writing to std::cout etc. since there's only one << buffer write for each thread

this sounds most reasonable.

parallel n_chains=8 n_threads=20

In cmdstanr these are specified like:

chains = 4,
parallel_chains = getOption("mc.cores", 1),
threads_per_chain = NULL,

Simplest thing is match that. If we skip the meta arg we avoid the extra parallel in parallel chains=4 parallel_chains=4.

I also like the rule that is search for filename X, if that isn't available, search for filename X1, X2, etc. where X1 and X2 two are rewrites of the name X. I'm not sure which direction is less confusing. We could add a check for ambiguity I guess.

[optional] I prefer .[chain_id] to _[chain_id].

So we're optionally broadcasting all the non-data inputs.

Rewriting the filenames on outputs also seems good.

That stringstream trick looks neat. Let's definitely do that.

@rok-cesnovar

chains = 4,
parallel_chains = getOption("mc.cores", 1),
threads_per_chain = NULL,

So at this level I think it's a bit different where for cmdstanr chains=4 they run 4 separate processes, parallel_chains is for R running those programs in parallel, and threads per chain is the amount of threads each program can use. I think here we might want something like

parallel chains=4 threads=16

Because threads here dictates the total number of threads available to cmdstan (a program could have parallel chains and each one uses a reduce_sum())

@rok-cesnovar should I also move the device and platform id options for OpenCL under parallel as well?

Yes, I also think we need to specify the total number of threads and that’s it...but we should do a quick benchmarkbto see if the tbb is indeed clever enough to run this optimal. So we do not want 8 cores used for one chain at a time, but instead 2 cores per chain when running 4 chains total.

@wds15

@wds15 yeah that's a good idea. fyi I just put up a PR to set the number of threads by an argument. Once that's in I can add the threads argument

stan-dev/math#2455

@rok-cesnovar

Yeah yeah, so in cmdstanr it is:

chains <- number of chains to run (not necessarily all of these in parallel)
parallel_chains <- maximum number of chains to work on at any time
threads_per_chain <- threads to use for each chain

I think here we might want something like

There and here will quickly be the same thing :D

So we do not want 8 cores used for one chain at a time, but instead 2 cores per chain when running 4 chains total.

The advantage of specifying total threads is that if there's only 1 chain left then all the threads can go there? Or once we switch to this do we actually lose the ability to even specify threads_per_chain? I'm okay switching I just want to understand the switch. Also @rok-cesnovar so he sees this.

If we do just num_chains and total_num_threads, then we somehow need to say that tasks corresponding to running a chain have higher priority for the threadpool than tasks generated by reduce sum. That sound right?

And then if we switch to chains/threads, then there isn't anymore a need for a parallel_chains? I guess the only need there would be if memory usage was a problem with the autodiff.

@wds15

The advantage of specifying total threads is that if there's only 1 chain left then all the threads can go there? Or once we switch to this do we actually lose the ability to even specify threads_per_chain?

Tmk threads_per_chain just tells each cmdstan program how many threads are available right? So like with one program idt we can tell the tbb how many threads we have per chain can we?

If we do just num_chains and total_num_threads, then we somehow need to say that tasks corresponding to running a chain have higher priority for the threadpool than tasks generated by reduce sum. That sound right?

@wds15 do you know if the tbb already gives higher priority to the top processes in nested parallelism? I assume it would

And then if we switch to chains/threads, then there isn't anymore a need for a parallel_chains? I guess the only need there would be if memory usage was a problem with the autodiff.

I think the arguments would change behavior where parallel chains would tell cmdstan to run X chains instead of right now where it runs X processes with one chain each.

Idt there would be any memory issues with running multiple chains (it should use less memory)

Depends on what are we going to do with parallel chains, all chains may be equal or there maybe a chain taking the lead. I think we should maintain flexibility by separating # of threads from # of chains.

As to args what I did for MPI cross chain is to have additional args under adapt meta-arg:
num_cross_chains, cross_chain_window, cross_chain_rhat, cross_chain_ess
and allow init arg to take regex file names, e.g
https://github.com/metrumresearchgroup/Torsten/blob/e50ca5e799ac29cb9d004e1791193072e4ad85d1/cmdstan/stan/src/stan/mcmc/cross_chain/mpi_cross_chain.hpp#L137

@wds15

@wds15 do you know if the tbb already gives higher priority to the top processes in nested parallelism? I assume it would

I don't know it... that's why I push for a quick test.

From a quick search I found this:

https://stackoverflow.com/questions/30589361/intel-tbb-disable-nested-parallelism

Which means to me we should try it out. In case performance is not good, we can control the concurrency level per chain by running each within a task_arena.

Depends on what are we going to do with parallel chains, all chains may be equal or there maybe a chain taking the lead. I think we should maintain flexibility by separating # of threads from # of chains.

Right now in the current PR I just want to do the embarrassingly parallel thing and run N chains in one program. But all the tooling around setting that up should make it easy to do auto warmup (and the code from here should be a good starting point for that).

As to args what I did for MPI cross chain is to have additional args under adapt meta-arg:
num_cross_chains, cross_chain_window, cross_chain_rhat, cross_chain_ess and allow init arg to take regex file names, e.g

Wrt the regex, how does the user know which init was used for which chain? I think I like the _i approach just so the user knows if something goofy happens with chain i they know it was caused by init_file_i

From a quick search I found this:

Oh yeah that is weird behavior. I'll try a benchmark today or tomorrow to see how this holds up

Wrt the regex, how does the user know which init was used for which chain? I think I like the _i approach just so the user knows if something goofy happens with chain i they know it was caused by init_file_i

I trust that in the end user won't be using it for a complex regex match but some simple pattern, such as xxx_1_xxx.json, xxx_2_xxx.json, ... or xxx_a_xxx.json, xxx_b_xxx.json, ..., similar to _i approach but with more flexibility. More importantly user is compelled to think what init files to select.

EDIT: also the output csv would have init file pointing to the correct file for a certain chain, right? So user don't need keep in mind the init file-chain map.

@wds15

should I also move the device and platform id options for OpenCL under parallel as well?

That was already released so we can not move it. In hindsight, I wish I would have gone with flattened opencl_device_id opencl_platform_id on the top level, the 2-level stuff makes everything just more confusing for the user as multilevel arguments are not common in command line tools (at least to my knowledge), but this is a topic for another time.

I think the parallel group is redundant as saying parallel threads=4 as opposed to just threads=4 brings no additional information, just more typing.

I think these are the options we have:

a) chains, threads_per_chain
b) num_chains, threads_per_chain
c) chains, threads
d) num_chains, threads
e) chains, cores
f) num_chains, cores

We do not want n_chains as we already use the num_ prefix in other arguments.

Options c) - f) set the overall number of threads, which defaults to chains/num_chains. Would also need an additional check that threads/cores >= chains. These two options have the benefit of using freed-up cores once other chains finish. We do need the benchmark @wds15 mentioned to see what TBB does and if its good.

Tmk threads_per_chain just tells each cmdstan program how many threads are available right?

Correct, cause that is the only thing we can do right now in cmdstanr/cmdstan. Whatever cmdstan will do, cmdstanr will follow, we do not have to overthink it. If running parallel chains this way will be 10,20 or 30% faster, we will want to always run parallel chains with a single cmdstan process, not via separate processes. parallel_chains in cmdstan will thus become redundant as Ben mentioned.

There is an additional angle of discussion here: STAN_THREADS flag.

Should STAN_THREADS=true become the default mode if this way of running parallel chains will be faster/more RAM efficient? This introduces a regression in the case of running a single chain without reduce_sum/map_rect (because of the use of thread safe lgamma and other functions and the different way of AD access).

I am guessing running multiple chains is what most users need/use, so this should be the default way? If a single chain is used, the users can disable threading if single chain performance is crucial to them.

For the record, my vote would currently go to c) or e) and setting threading on by default, but I am not particularly confident in my votes (pending benchmarks & additional discussion).

I like c and always threaded. That assumes the chain/reduce_sum task priority isn't a problem, and that we don't eat too much loss with the lgamma (is there a long list of these? or is it just lgamma)?

The question of threading should be on always is hard! This needs to consider how this affects performance on all platforms which is really different (windows!).

The best would be to just specify the number of threads and that’s it, so c...but that needs a benchmark. What about supporting per thread and thread total as options? Is that a consideration?

@wds15

I think the parallel group is redundant as saying parallel threads=4 as opposed to just threads=4 brings no additional information, just more typing.

Yeah I think that's fair we can just move it to threads

Options c) - f) set the overall number of threads, which defaults to chains/num_chains. Would also need an additional check that threads/cores >= chains. These two options have the benefit of using freed-up cores once other chains finish. We do need the benchmark @wds15 mentioned to see what TBB does and if its good.

What's the reason behind making sure threads/cores >= chains? If someone only has 4 threads and 2 cores and wants 4 chains is there something wrong with them going in batches of 2?

Should STAN_THREADS=true become the default mode if this way of running parallel chains will be faster/more RAM efficient? This introduces a regression in the case of running a single chain without reduce_sum/map_rect (because of the use of thread safe lgamma and other functions and the different way of AD access).

I am guessing running multiple chains is what most users need/use, so this should be the default way? If a single chain is used, the users can disable threading if single chain performance is crucial to them.

Yeah oof that is a tough one. imo I think the answer is probably yes people will normally do more than one chain at once. But until I have time to do benchmarking idt I'm going to have a definite vote on this either.

I'm fine with (c), though I have a slight preference for (d) because I like that they all sort of match like num_warmup num_sample num_chains, but that's a small personal preference and (c) is totally fine for me

Navigation Menu

Repository navigation

How to do IO args for parallel cmdstan? #1002

Summary:

New args

Multiple IO

Current Version:

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

How to do IO args for parallel cmdstan? #1002

Description

Summary:

New args

Multiple IO

Current Version:

Activity

mitzimorris commented on Mar 26, 2021

SteveBronder commented on Mar 26, 2021

SteveBronder commented on Mar 26, 2021

mitzimorris commented on Mar 26, 2021

bbbales2 commented on Mar 30, 2021

SteveBronder commented on Mar 31, 2021

wds15 commented on Mar 31, 2021

SteveBronder commented on Mar 31, 2021

bbbales2 commented on Mar 31, 2021

bbbales2 commented on Mar 31, 2021

SteveBronder commented on Mar 31, 2021

yizhang-yiz commented on Mar 31, 2021

wds15 commented on Mar 31, 2021

SteveBronder commented on Mar 31, 2021

yizhang-yiz commented on Mar 31, 2021

rok-cesnovar commented on Apr 1, 2021

bbbales2 commented on Apr 1, 2021

wds15 commented on Apr 1, 2021

SteveBronder commented on Apr 4, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions