Sixdesk Tools

Stages

In this module the stages are organized.

class pylhc_submitter.sixdesk_tools.stages.CheckInput(jobname: str, jobargs: dict, env: AutoSixEnvironment)

Check if input files have been generated properly > cd $basedir/workspace-$jobname/sixjobs > /afs/cern.ch/project/sixtrack/SixDesk_utilities/pro/utilities/bash/mad6t.sh -c

If not, and resubmit is active > /afs/cern.ch/project/sixtrack/SixDesk_utilities/pro/utilities/bash/mad6t.sh -w

class pylhc_submitter.sixdesk_tools.stages.CheckSixtrackOutput(jobname: str, jobargs: dict, env: AutoSixEnvironment)

Checks sixtrack output via run_status. If this fails even though all jobs have finished on the scheduler, check the log-output (run_status messages are logged to debug). > cd $basedir/workspace-$jobname/sixjobs > /afs/cern.ch/project/sixtrack/SixDesk_utilities/pro/utilities/bash/run_status

If not, and resubmit is active > cd $basedir/workspace-$jobname/sixjobs > /afs/cern.ch/project/sixtrack/SixDesk_utilities/pro/utilities/bash/run_six.sh -i

class pylhc_submitter.sixdesk_tools.stages.CreateJob(jobname: str, jobargs: dict, env: AutoSixEnvironment)

create workspace > cd $basedir > /afs/cern.ch/project/sixtrack/SixDesk_utilities/pro/utilities/bash/set_env.sh -N workspace-$jobname

write sixdeskenv, sysenv, filled mask (manual)

class pylhc_submitter.sixdesk_tools.stages.Final(jobname: str, jobargs: dict, env: AutoSixEnvironment)

Just info about finishing this script and where to check the stagefile.

class pylhc_submitter.sixdesk_tools.stages.InitializeWorkspace(jobname: str, jobargs: dict, env: AutoSixEnvironment)

initialize workspace > cd $basedir/workspace-$jobname/sixjobs > /afs/cern.ch/project/sixtrack/SixDesk_utilities/pro/utilities/bash/set_env.sh -s

remove the twiss-fail check in sixtrack_input (manual)

class pylhc_submitter.sixdesk_tools.stages.PostProcess(jobname: str, jobargs: dict, env: AutoSixEnvironment)

Extracts the analysed data in the database and writes them to three tfs files:

• All DA values

• Statistics over angles, listed per seed (+ Seed 0 as over seeds and angles)

• Statistics over seeds, listed per angle

The statistics over the seeds are then plotted in a polar plot. All files are outputted to the sixjobs/autosix_output folder in the job directory.

class pylhc_submitter.sixdesk_tools.stages.SixdbCmd(jobname: str, jobargs: dict, env: AutoSixEnvironment)

Analysise results in database via sixdb. > cd $basedir/workspace-$jobname/sixjobs > python3 /afs/cern.ch/project/sixtrack/SixDesk_utilities/pro/utilities/externals/SixDeskDB/sixdb $jobname da

when fixed: > python3 /afs/cern.ch/project/sixtrack/SixDesk_utilities/pro/utilities/externals/SixDeskDB/sixdb $jobname da_vs_turns -turnstep 100 -outfile > python3 /afs/cern.ch/project/sixtrack/SixDesk_utilities/pro/utilities/externals/SixDeskDB/sixdb $jobname plot_da_vs_turns

class pylhc_submitter.sixdesk_tools.stages.SixdbLoad(jobname: str, jobargs: dict, env: AutoSixEnvironment)

Gather results into database via sixdb. > cd $basedir/workspace-$jobname/sixjobs > python3 /afs/cern.ch/project/sixtrack/SixDesk_utilities/pro/utilities/externals/SixDeskDB/sixdb . load_dir

class pylhc_submitter.sixdesk_tools.stages.Stage(jobname: str, jobargs: dict, env: AutoSixEnvironment)

Abstract Base Class for all Stages. Also provides the basic methods. The stages themselves only need to implement the _run() method.

class pylhc_submitter.sixdesk_tools.stages.StageMeta(name, bases, namespace, /, **kwargs)

Dynamically generate name and value from STAGE_ORDER

class pylhc_submitter.sixdesk_tools.stages.SubmitMask(jobname: str, jobargs: dict, env: AutoSixEnvironment)

submit for input generation > cd $basedir/workspace-$jobname/sixjobs > /afs/cern.ch/project/sixtrack/SixDesk_utilities/pro/utilities/bash/mad6t.sh -s

class pylhc_submitter.sixdesk_tools.stages.SubmitSixtrack(jobname: str, jobargs: dict, env: AutoSixEnvironment)

Generate simulation files (-g) and check if runnable (-c) and submit (-s) (-g -c -s == -a). > cd $basedir/workspace-$jobname/sixjobs > /afs/cern.ch/project/sixtrack/SixDesk_utilities/pro/utilities/bash/run_six.sh -a

Create SixDesk Workspace

Tools to setup the workspace for sixdesk.

pylhc_submitter.sixdesk_tools.create_workspace.create_job(jobname: str, basedir: Path, executable: Path | str, mask_text: str, sixdesk: Path = PosixPath('/afs/cern.ch/project/sixtrack/SixDesk_utilities/pro'), ssh: str = None, **kwargs)

Create environment and individual jobs/masks for SixDesk to send to HTC.

Keyword Arguments:

mask. (Need to contain all replacements for sixdeskenv and the)

pylhc_submitter.sixdesk_tools.create_workspace.fix_pythonfile_call(jobname: str, basedir: Path)

Removes ‘<’ in the binary file line in mad6t.sh so __file__ works.

pylhc_submitter.sixdesk_tools.create_workspace.init_workspace(jobname: str, basedir: Path, sixdesk: Path = PosixPath('/afs/cern.ch/project/sixtrack/SixDesk_utilities/pro'), ssh: str = None)

Initializes the workspace with sixdeskenv and sysenv.

pylhc_submitter.sixdesk_tools.create_workspace.remove_twiss_fail_check(jobname: str, basedir: Path)

Comments out the “Twiss fail” check from mad6t.sh

pylhc_submitter.sixdesk_tools.create_workspace.set_max_materialize(sixdesk: Path, max_materialize: int = None)

Adds the max_materialize option into the htcondor sixtrack submission-file.

SixDesk Submission Utils

Individual functions to call SixDesk functionality.

pylhc_submitter.sixdesk_tools.submit.check_sixtrack_input(jobname: str, basedir: Path, sixdesk: Path = PosixPath('/afs/cern.ch/project/sixtrack/SixDesk_utilities/pro'), ssh: str = None, resubmit: bool = False)

Checks the generated input files needed by sixtrack and resubmits, if requested.

pylhc_submitter.sixdesk_tools.submit.check_sixtrack_output(jobname: str, basedir: Path, python: Path | str, sixdesk: Path = PosixPath('/afs/cern.ch/project/sixtrack/SixDesk_utilities/pro'), ssh: str = None, resubmit: bool = False)

Checks if the sixtrack output is all there.

pylhc_submitter.sixdesk_tools.submit.sixdb_cmd(jobname: str, basedir: Path, python: Path | str, cmd: list, sixdesk: Path = PosixPath('/afs/cern.ch/project/sixtrack/SixDesk_utilities/pro'), ssh: str = None)

Performs analysis on the sixdb database.

pylhc_submitter.sixdesk_tools.submit.sixdb_load(jobname: str, basedir: Path, python: Path | str, sixdesk: Path = PosixPath('/afs/cern.ch/project/sixtrack/SixDesk_utilities/pro'), ssh: str = None)

Creates sixdb database and loads the study results into it.

pylhc_submitter.sixdesk_tools.submit.submit_mask(jobname: str, basedir: Path, sixdesk: Path = PosixPath('/afs/cern.ch/project/sixtrack/SixDesk_utilities/pro'), ssh: str = None)

Run the mask (probably Madx) and generate sixtrack input files.

Parameters:

• jobname (str) -- Name of the Job

• basedir (Path) -- SixDesk Basefolder Location

• sixdesk (Path) -- Path to the SixDesk installation/repo

pylhc_submitter.sixdesk_tools.submit.submit_sixtrack(jobname: str, basedir: Path, python: Path = None, sixdesk: Path = PosixPath('/afs/cern.ch/project/sixtrack/SixDesk_utilities/pro'), ssh: str = None, resubmit: bool = False)

Generate simulation files and check if runnable and submit.

Post Process DA

Tools to process data after sixdb has calculated the da. Includes functions for extracting data from database as well as plotting of DA polar plots.

pylhc_submitter.sixdesk_tools.post_process_da.create_da_tfs(jobname: str, basedir: Path) → Tuple[TfsDataFrame, TfsDataFrame, TfsDataFrame]

Extracts data from db into dataframes, and writes and returns them.

Parameters:

• jobname (str) -- Name of the Job

• basedir (Path) -- SixDesk Basefolder Location

pylhc_submitter.sixdesk_tools.post_process_da.create_polar_plots(jobname: str, basedir: Path, df_da: TfsDataFrame, df_angles: TfsDataFrame)

Plotting loop over da-methods and wrapper so save plots.

Parameters:

• jobname (str) -- Name of the Job

• basedir (Path) -- SixDesk Basefolder Location

• df_da (TfsDataFrame) -- Full DA analysis result.

• df_angles (TfsDataFrame) -- Dataframe with the statistics (min, max, mean) per angle

pylhc_submitter.sixdesk_tools.post_process_da.plot_polar(df_angles: TfsDataFrame, da_col: str = 'ALOST2', jobname: str = '', df_da: TfsDataFrame = None, **kwargs) → Figure

Create Polar Plot for DA analysis data.

Keyword arguments are all optional.

Parameters:

• df_angles (TfsDataFrame) -- Dataframe with the statistics (min, max, mean) per angle

• da_col (str) -- DA-Column name from sixdesk analysis to be used , e.g. ALOST2. (optional, default: ALOST2)

• jobname (str) -- Name of the job. Used in window title only (optional).

• df_da (TfsDataFrame) -- Full DA analysis result. If given, plots the individual DA results per seed. (optional)

Keyword Arguments:

• interpolated (bool) -- If true, uses interpolation to plot the lines curved

• fill (bool) -- If true, fills the area between min and max with light blue

• angle_ticks (Iterable[numeric]) -- Positions in degree of the angle ticks (and lines)

• ticks (amplitude) -- Positions of the amplitude ticks.

Returns:

Figure of the polar plot.

pylhc_submitter.sixdesk_tools.post_process_da.post_process_da(jobname: str, basedir: Path)

Post process the DA results into dataframes and DA plots.

Extract Data From DataBase

These functions operate on the SixDesk database and help to extract data form it.

TODO: Implement extraction of data into .csv (and/or tfs?) like fvanderv does.

pylhc_submitter.sixdesk_tools.extract_data_from_db.extract_da_data(jobname: str, basedir: Path) → TfsDataFrame

Extract DA data directly from the database.

Parameters:

• jobname (str) -- Name of the Job

• basedir (Path) -- SixDesk Basefolder Location

pylhc_submitter.sixdesk_tools.extract_data_from_db.extract_meta_data(jobname: str, basedir: Path) → TfsDataFrame

Extract the meta-data directly from the database.

Parameters:

• jobname (str) -- Name of the Job

• basedir (Path) -- SixDesk Basefolder Location

SixDesk Utilities

Helper Utilities for Autosix.

pylhc_submitter.sixdesk_tools.utils.check_mask(mask_text: str, replace_args: dict)

Checks validity/compatibility of the mask and replacement dict.

pylhc_submitter.sixdesk_tools.utils.is_locked(jobname: str, basedir: Path, unlock: bool = False)

Checks for sixdesklock-files

SixDesk Troubleshooting tools

Some useful functions to troubleshoot the SixDesk output.

pylhc_submitter.sixdesk_tools.troubleshooting.check_last_stage(jobname: str, basedir: Path)

Logs names of all last run stages for given jobs.

pylhc_submitter.sixdesk_tools.troubleshooting.check_sixtrack_output_data(jobname: str, basedir: Path)

Presence checks for SixDesk tracking output data.

This checks recursively all directories in track. Will be busy for a while.

pylhc_submitter.sixdesk_tools.troubleshooting.check_stages_for_setup(basedir: Path, stage_name: str, jobid_mask: str, replace_dict: dict)

Check the last run stage for all jobs from given job-setups.

pylhc_submitter.sixdesk_tools.troubleshooting.find_obviously_failed_sixtrack_submissions(basedir: Path)

Checks in jobs in track whether the directory structure seems to be created and if there is output data. This checks only the first directories found, to speed up this process. For a more precise scan see check_sixtrack_output_data.

pylhc_submitter.sixdesk_tools.troubleshooting.for_all_jobs(func: callable, basedir: Path, *args, **kwargs)

Do function for all jobs in basedir.

pylhc_submitter.sixdesk_tools.troubleshooting.get_all_jobs_in_base(basedir)

Returns all job-names in the sixdeskbase dir.

pylhc_submitter.sixdesk_tools.troubleshooting.get_first_dir(cwd: Path, glob: str = '*')

Return first directory of pattern glob.

pylhc_submitter.sixdesk_tools.troubleshooting.get_last_stage(jobname, basedir)

Get the last run stage of job jobname.

pylhc_submitter.sixdesk_tools.troubleshooting.reset_stages(jobname: str, basedir: Path, stage_name: str)

Reset stages until stagename, i.e. similar to set_stages but only if the stage has already been run. Inverse to skip_stages

pylhc_submitter.sixdesk_tools.troubleshooting.set_stages(jobname: str, basedir: Path, stage_name: str)

Sets the last run stage of all given jobs to stage_name.

pylhc_submitter.sixdesk_tools.troubleshooting.set_stages_for_setup(basedir: Path, stage_name: str, jobid_mask: str, replace_dict: dict)

Sets the last run stage for all jobs from given job-setups.

pylhc_submitter.sixdesk_tools.troubleshooting.skip_stages(jobname: str, basedir: Path, stage_name: str)

Skip stages until stagename, i.e. similar to set_stages but only if the stage hasn’t been reached yet. Inverse to reset_stages