Message: Saving partial results Not Logged In (login)
 Next-in-Thread Next-in-Thread
 Next-in-Forum Next-in-Forum

None Saving partial results 

Forum: Multithreading
Date: 18 Aug, 2015
From: <herr_apa>

Hi,

I'm running a simulation that includes movement. In my GeneratePrimaries, I offset the position of the initial particle by an amount that depends on the event number. The position of the particle is x(t), and t = N/Ntot where N is the event number of the current event and Ntot is the total number of events to be simulated. The quantity of interest is the energy deposited in different parts of the target, which I record using a G4PSEnergyDeposit.

I would like to make a "movie" of how the dose distribution develops as a function of time. That is, I want to save the output of the simulation every so many events. For the sake of example, let's say I'm simulating 1000 events and I want a snapshot of the output every 50. In single-threaded mode, I could simply make an EndOfEventAction to check the current event number N, and if (N % 50) == 0 I could fetch the deposited energy from my G4THitsMap and write it to file. In multithreaded mode this approach is not possible because each thread only has access to the events it happened to process, and only one of the threads will process the event that triggers the write.

Question 1: When a thread detects that (N % 50) == 0, is it possible to somehow merge all the events processed so far across all threads so that the output can be written to file?

Another approach could be to tell Geant4 "I want to simulate 1000 events - But start the run at event 400 and stop it at event 449!" That way the time dependence and merging would happen as normal, and I could just split the run into many sub runs that start and stop at the appropriate event numbers. The output would not be cumulative for each segment of the run like with the first approach, but this could be fixed by post-processing.

Question 2: Is there a convenient way to process only a subrange of the events in a run? Maybe a check in BeginOfEventAction to skip the entire event unless it is in the desired range. Can this be done?

Suggestions on other approaches are also welcome. I might very well be missing something obvious.

// Anders

Inline Depth:
 0 0
 1 1
Add message: (add)

Feedback Re: Saving partial results  by Andrea Dotti <Andrea Dotti>,   26 Aug, 2015
Re: None Saving partial results
Hello, sorry for the late answer.

> 
> Question 1: When a thread detects that (N % 50) == 0, is it possible to
> somehow merge all the events processed so far across all threads so that
> the output can be written to file?
> 

No this is not possible, because this would require synchronization between threads during event loop that would cause a large CPU penalty.

> 
> Question 2: Is there a convenient way to process only a subrange of the
> events in a run? Maybe a check in BeginOfEventAction to skip the entire
> event unless it is in the desired range. Can this be done?

I think you have already found the correct solution to your problem. The solution is to make separate runs. A G4UserRunAction should be used. For example: t=Nevt/Ntot + Ntot*Nrun where Nrun is the run number.

Geant4 will guarantee that G4Run are correctly merged and threads are synchronized between the runs that compose your job.

Andrea

None Re: Saving partial results  by <herr_apa>,   20 Aug, 2015
Re: None Saving partial results
Hi again,

I've worked some more on this problem, and now I have a different question: In multithreaded mode, how can I prompt a call to G4VUserActionInitialization::Build after having already called beamOn once?

And here's some context if that helps. Let's go back to the 1000 events that I wanted to split up into 50-event slices. What I do now is loop over beamOn and execute twenty runs 50 events each. The position of the initial particle is calculated by a dedicated class called ParticlePosition, that defines x(t). I added two new variables to this class: The total number of events to be simulated, Nsim (1000 in this example), and the number of events completed by previous runs, Noffset. Time then becomes t = (Noffset + N)/Nsim where N is the event number of the current event (i.e. a number between 0 and 49 in this example). Before each call to beamOn, I update Noffset. And this is where I run into trouble.

The program is structured such that an ActionInitialization is created in the main function. This ActionInitialization has a ParticlePosition variable. In ActionInitialization::Build a PrimaryGeneratorAction is created, and the ParticlePosition is passed to its constructor. Everything works fine the first time beamOn is called. For each subsequent turn in the loop, I update the Noffset in the ParticlePosition owned by the ActionInitialization and call runManager->SetUserInitialization(actionInitialization) before the next call to beamOn. Had it been a single-threaded G4RunManager, Build would have been invoked. But because it is a G4MTRunManager, only BuildForMaster is invoked. No new PrimaryGeneratorAction object is created, because that happens in Build. How can I relay the updated ParticlePosition to my PrimaryGeneratorAction? Is there a way to destroy the worker threads and then re-create them (which would invoke Build)?

Thanks in advance!

// Anders

Warning Re: Saving partial results  by Andrea Dotti <Andrea Dotti>,   26 Aug, 2015
Re: None Re: Saving partial results
Hello again,

On Thu, 20 Aug 2015 14:38:48 GMT, herr_apa wrote:

> Hi again,
> 
> I've worked some more on this problem, and now I have a different
> question: In multithreaded mode, how can I prompt a call to
> G4VUserActionInitialization::Build after having already called beamOn
> once?

You cannot. This is a kernel function being called by RunManager when thread need initialization (once and only once). Consider using setters/getters to add the functionality that you need. If you need to change something between runs, you can for example use a Setter method called by your RunAction.

> 
> And here's some context if that helps. Let's go back to the 1000 events
> that I wanted to split up into 50-event slices. What I do now is loop
> over beamOn and execute twenty runs &#65533; 50 events each. The position of
> the initial particle is calculated by a dedicated class called
> ParticlePosition, that defines x(t). I added two new variables to this
> class: The total number of events to be simulated, Nsim (1000 in this
> example), and the number of events completed by previous runs, Noffset.
> Time then becomes t = (Noffset + N)/Nsim where N is the event number of
> the current event (i.e. a number between 0 and 49 in this example).
> Before each call to beamOn, I update Noffset. And this is where I run
> into trouble.
> 
> The program is structured such that an ActionInitialization is created
> in the main function. This ActionInitialization has a ParticlePosition
> variable. In ActionInitialization::Build a PrimaryGeneratorAction is
> created, and the ParticlePosition is passed to its constructor.
> Everything works fine the first time beamOn is called. For each
> subsequent turn in the loop, I update the Noffset in the
> ParticlePosition owned by the ActionInitialization and call
> runManager->SetUserInitialization(actionInitialization) before the next
> call to beamOn. Had it been a single-threaded G4RunManager, Build would
> have been invoked. But because it is a G4MTRunManager, only
> BuildForMaster is invoked. No new PrimaryGeneratorAction object is
> created, because that happens in Build. How can I relay the updated
> ParticlePosition to my PrimaryGeneratorAction? Is there a way to destroy
> the worker threads and then re-create them (which would invoke Build)?
> 

What you are doing is not a good idea. You should never call SetUserInitialization methods more than once. Again it is better that you add functionality to your derived user-action classes to modify internal parameters as you need. Also, currently, threads are a static pool and cannot be re-created.

I may be misinterpreting something here, but I think all your needs can be taken into account creating different user-actions. The goal of user-actions is to allow for changing the behavior of your application while keeping a clean and correct design.

Andrea

> Thanks in advance!
> 
> // Anders
> 

None Re: Saving partial results  by <herr_apa>,   25 Aug, 2015
Re: None Re: Saving partial results
In the end, I solved the problem by making ParticlePosition aware of the run number. In my BeginOfRunAction, I have each of my worker threads access their PrimaryGeneratorAction and set the run number in their thread-local ParticlePosition. Then the class can calculate Noffset on its own.

It would still be interesting to get answers to my earlier questions for completion.

// Anders

 Add Message Add Message
to: "Saving partial results"

 Subscribe Subscribe

This site runs SLAC HyperNews version 1.11-slac-98, derived from the original HyperNews