WHIZARD

Very long simulation time based on existing integration

Asked by Ulrike Schnoor

Dear Whizards,

I am trying to generate large event samples of ee->qqqqlnu for all combinations of flavors with Whizard 2.8.3.

A sindarin file and corresponding log file can be found here:
https://cernbox.cern.ch/index.php/s/k46yDQLMEXbbDXY

Since the integration takes more than a day, for the production of large event samples I wanted to do the integration once, and then ship the integration files to the grid. There, whizard is run again, so it reads in the available files and then generates events based on those. However, surprisingly, the event generation takes very long. I have reproduced this locally, to make sure it is not related to the distributed setup. The integration is read in quickly (5-10 min), but then, it takes very long before the actual simulation starts. From what I can see, the grid files are read in, then the libraries for each subprocess (there are many) are created. Next, it seems to be initializing the integration again - although it has been done before. This takes very long because there are many subprocesses. Finally, just before starting the actual event simulation, there is a certain time in which it just prints this message multiple times:
Warning: CIRCE2: User-defined beam polarization set for unpolarized CIRCE2 data file

Do you have any advice how to reduce the time it takes to simulate events based on existing integration files? I have tried the same (i.e. reading in existing integration files and generating 25 events) with diagonal CKM matrix, and this takes only one hour whereas with the physical CKM matrix it takes 8-9 hours. So I assume it is connected to the huge number of subchannels. The diagonal CKM matrix would certainly be a workaround for me, but I would like to know the reason and if there might be another way to speed it up.

A related feature request: Would it be possible to introduce an option to write timestamps into the log file?

Thank you,
Ulrike

Question information

Language:
English Edit question
Status:
Solved
For:
WHIZARD Edit question
Assignee:
Simon Braß Edit question
Solved by:
Simon Braß
Solved:
Last query:
Last reply:
Revision history for this message
Best Simon Braß (sbrass) said : 		#1

Dear Ulrike,

your description fits a simulation with `?resonance_history = true`.

During the setup of the simulation, WHIZARD sets up the possible intermediate (sub-)processes, which each lead to the final state of the overall process definition, by creating and compiling their appropriate matrix element code into a library, loading the library and doing a *full* integration on each subprocess.
However, WHIZARD does not go through this setup during integration, only for a `simulate`, hence, you see the additional computation during your simulation step.
Behold, however, that is, so far, my understanding of the implementation of the resonance history feature - Wolfgang may comment on this as he came up with it and implemented it.

As (intermediate) solution, I would suggest that you let a mock simulation run right after your initial integration with a single event (`n_events = 1`), in order to force WHIZARD to produce the subprocesses library and the integration grids.
You can then pack all the generated files, and distribute them among your grid nodes, such that, the single invocations of WHIZARD do not need to run the setup for the resonance histories, and the actual event simulation starts directly.

Cheers,
Simon

Revision history for this message
Ulrike Schnoor (ulrike-sch) said : 		#2

Dear Simon,

thank you for the quick reply!

That makes sense to me and indeed, the processes that are initialized for the simulation seem to be the ones with W and Z intermediate states, so that could have tipped me off that it has to do with the resonance history.

I quickly checked and it does seem that indeed the whole run takes somewhat less time. I will check in more detail.

I'd want to be sure that the one generated event is not replicated in each of the subjobs. What is the best way to do that? I could just not ship the .evx file, for example.

Cheers,
Ulrike

Revision history for this message
Simon Braß (sbrass) said : 		#3

Dear Ulrike,

thanks for checking.

You are correct with the .evx file, which is just a binary dump of the internal event representation, e.g. allows for simple restarts, rescans and so on.
If you remove it, then WHIZARD has "no" memory of the events anymore.

Cheers,
Simon

Revision history for this message
Ulrike Schnoor (ulrike-sch) said : 		#4

Thanks a lot, I will proceed as you suggested.

Ulrike

Revision history for this message
Ulrike Schnoor (ulrike-sch) said : 		#5

Thanks Simon Braß, that solved my question.