pySAS Classes and Functions

ObsID

Configure pySAS

class pysas.config_pysas.sas_config(config_file=None)[source]

Bases: object

Class for interacting with the pySAS configuration file.

Parameters:

config_file (str, optional) – Path to config file, by default ‘XDG_CONFIG_HOME (usually the user’s HOME directory).

get_setting(option, section='sas')[source]

Retrieves a setting from the current session configuration (not from the config file).

Parameters:

  • option (str) – Which setting to retrieve.

  • section (str, optional) – Which section in the config file, by default ‘sas’.

Returns:

The returned setting.

Return type:

str

set_setting(option, value, section='sas')[source]

Sets a setting in the current session configuration (does not change the config file).

Parameters:

  • option (str) – Which setting to set.

  • value (str) – Value to be set.

  • section (str, optional) – Which section in the configuration has the option, by default ‘sas’.

set_setting_and_save(option, value, section='sas')[source]

Sets a setting in the current session configuration AND saves the setting to the config file.

Parameters:

  • option (str) – Which setting to set.

  • value (str) – Value to be set.

  • section (str, optional) – Which section in the configuration has the option, by default ‘sas’.

save_config(config_file_path=None)[source]

Saves the current session configuration to the config file. Does not modify any configuration options.

Parameters:

config_file_path (str, optional) – Path to config file, by default None.

show_current_config()[source]

Prints the current session configuration settings to the terminal.

Note: The current session configuration settings are usually the same as the contents of the config file, but they can be different.

show_config_file()[source]

Prints the contents of the configuration file to the terminal.

reset_to_defaults()[source]

Resets configuration file to the defaults.

simple_config(sas_dir=None, sas_ccfpath=None, data_dir=None, repo=None)[source]

For quick, simple configuration of pySAS.

Parameters:

  • sas_dir (str, optional) – Directory where SAS is installed, by default None.

  • sas_ccfpath (str, optional) – Directory with calibration files, by default None.

  • data_dir (str, optional) – User data directory, by default None.

  • repo (str, optional) –

    Default repository for downloading data, by default None. Accepted values are,

    ’ESA’ (data from the XSA)

    ’NASA’ (data from the HEASARC)

    ’AWS’ (data from AWS s3 bucket (NASA))

    ’Fornax’ (if user is on Fornax)

    ’SciServer’ (if user is on SciServer)

pysas.config_pysas.run_config()[source]

The purpose of this script is so that the user can set the pySAS defaults

sas_dir (required)

sas_ccfpath (required)

data_dir (optional)

Once the defaults are set by the user, SAS will automatically be initialized when pySAS is imported (import pysas).

The user can also optionally set a default data directory (data_dir) where observation data files (odf) will be downloaded. A separate subdirectory will be made for each observation ID (Obs ID).

The default data directory can be set or change later using functions in the ‘sas_cfg’ object.

For example:

import pysas
data_path = '/path/to/data/dir/'
pysas.sas_cfg.set_setting_and_save('data_dir', data_path)

The default values for the SAS directory (sas_dir), the path to the calibration files (sas_ccfpath), along with ‘verbosity’ and ‘suppress_warning’, can also be set in the same way.

At any time the user can reset the config file to the defaults,:

import pysas
pysas.sas_cfg.reset_to_defaults()

MyTask Class

All stand alone SAS tasks can be called by using MyTask. The task invocation command may include several parameters and generic options.

class pysas.sastask.MyTask(taskname, inargs={}, logfilename=None, tasklogdir=None, output_to_terminal=True, output_to_file=False, logger=None)[source]

Bases: object

Class for SAS tasks.

When a class instance is generated the corresponding parameter file is read. If input arguments are passed in as a list or string then they are converted into a dictionary, but input arguments are not processed until either ‘run’ or ‘processargs’ is called.

Parameters:

  • taskname (str) – SAS task name.

  • inargs (dict | list | str, optional) – SAS input arguments, by default {}.

  • logfilename (str, optional) – Designated log file name. Useful for putting all output from multiple tasks into the same file. By default “{taskname}.log”.

  • tasklogdir (str, optional) –

    Output directory for the log file, by default None. Priority of defaults for task_logdir

    1. tasklogdir (passed in)

    2. SAS_TASKLOGDIR (envirnment variable)

    3. current directory (final default)

  • output_to_terminal (bool, optional) – Whether to print output to the terminal, by default True.

  • output_to_file (bool, optional) – Whether to print output to file, by default False.

  • logger (logger, optional) – Logger object, by default None.

readparfile()[source]

Reads the parameter file (taskname.par) for the task. Adds various lists and dictionaries to the object.

allparams (dictionary)

Dictionary of the form: { parameter: {attributes} }

mandpar (list)

List of all mandatory parameters/subparameters

mainparams (list)

List of main paramaters

parmap (list)

Dictionary of the form: { pname: [ subparameters …], …}

mandpar_dict (dictionary)

Dictionary, key = mandatory subparamater, value=its parent parameter

rev_mandpar_dict (dictionary)

Dictionary, keys = parent paramater, value = list of all mandatory subparameters

rev_mandpar_string_dict (dictionary)

Dictionary, keys = parent parameter type ‘string’, value = list of alternatives

processargs()[source]

Processes the input arguments and compares them with the parameter file.

Returns:

Whether to exit without running the task based on the inputs.

Return type:

bool

Raises:

  • Exception – Parameter ‘{p}’ is not recognized!

  • Exception – Missing, at least, mandatory parameter “{p}”.

  • Exception – If subparameter {p} is used then {parent} must be set to “{cond_par_val}”!

  • Exception – Missing mandatory subparameter {child}.

printHelp()[source]

Wrapper for paramXmlInfo.printHelp. Prints information on the SAS task.

run()[source]

Runs the SAS task with the inputs stored in the dictionary iparsdic.

SAS tasks are invoked through a subprocess call.

Returns:

Whether to exit without running the task based on the inputs.

Return type:

bool

Raises:

EnvironmentError – SAS_PATH is undefined!

runtask()[source]

This method is here for legacy reasons since some Python based SAS tasks still use ‘runtask’ instead of ‘run’.

class pysas.sastask.SASTask(taskname, inargs)[source]

Bases: MyTask

Class SASTask is a child of MyTask and has access to all of the methods in MyTask. It’s only purpose is to act as a placeholder for legacy code.

Initalize SAS Functions

For auto initializing SAS on import of pySAS.

pysas.init_sas.add_environ_variable(variable, invalue, prepend=True)[source]

The function first checks if the enviroment variable already exists. If not it will be created and set to value. If variable alread exists the function will check if value is already present. If not it will add it either prepending (default) or appending.

Parameters:

  • variable (str) – The name of the environment variable to be set.

  • invalue (str | list) – The value to which the environment variable will be set.

  • prepend (bool, optional) – whether to prepend or append the variable, by default True (prepend).

Raises:

TypeError – Input value must be str or list.

pysas.init_sas.overwrite_environ_variable(variable, invalue)[source]

Will overwrite an environment variable to a new value. Will not check if variable exists or if it is defined.

Parameters:

  • variable (str) – The name of the environment variable to be set.

  • invalue (str | list) – The value to which the environment variable will be set.

pysas.init_sas.initializesas(sas_dir, sas_ccfpath, verbosity=4, suppress_warning=1, image_viewer='ds9')[source]

Initializes SAS. Heasoft must be initialized first, separately.

Parameters:

  • sas_dir (str | Path) – Directory where SAS is installed.

  • sas_ccfpath (str | Path) – Directory where calibration files are located.

  • verbosity (int, optional) – SAS verbosity, by default 4.

  • suppress_warning (int, optional) – SAS suppress warning, by default 1.

  • image_viewer (str, optional) – SAS image viewer, by default ‘ds9’.

Returns:

Information about SAS envirment veriables that were set.

Return type:

str

Raises:

  • EnvironmentError – LHEASOFT is not set. Please initialise HEASOFT.

  • Exception – HEASOFT is not initialized. Please initialise HEASOFT.

  • NotADirectoryError – sas_dir must be provided to initialize SAS.

  • NotADirectoryError – sas_ccfpath must be provided to initialize SAS.

pySAS Logging

Module for logging pySAS output.

pysas.logger.get_logger(taskname, toterminal=True, tofile=False, logfilename=None, tasklogdir=None, pylogger=True)[source]

Function to get a logger object. Uses the package ‘loguru’.

Parameters:

  • taskname (str) – Name of the task to be run. By default will name log file “{taskname}.log”.

  • toterminal (bool, optional) – Output will be written to the terminal, by default True.

  • tofile (bool, optional) – Output will be written to a log file, by default False.

  • logfilename (str, optional) – Designated log file name. Will be used instead of “{taskname}.log”. Useful for putting all output from multiple tasks into the same file, by default “{taskname}.log”.

  • tasklogdir (str, optional) – Directory where to write the log file. Priority of defaults for task_logdir are; tasklogdir (passed in to function), SAS_TASKLOGDIR (envirnment variable), then cwd (final default).

  • pylogger (bool, optional) – True: Use logger settings for Python logging False: Use logger settings for subprocess logging Only set to ‘False’ in sastask.py -> MyTask.run() when running a non Python SAS task using subprocess, by default True.

Returns:

Logger instance.

Return type:

logger

class pysas.logger.TaskLogger(taskname)[source]

Bases: object

This is only here for legacy reasons. This was the original logger class for pySAS.

TaskLogger creates a logger for Python tasks. The name of the logger is the name of the task. Such logger is known as saslogger. It includes two handlers, one console handler and one file handler. The file handler will put all logging messages in a file named <taskname>.log.

This file is by default written in the starting directory of the task (os.getcwd()). However, for each task we can set that directory by taking into account the env variable SAS_TASKLOGDIR, if defined.

By default the logging file is created in mode -a- (append) to allow to use the same file to log several runs of the same task. However, the SAS_TASKLOGFMODE environment variable can be set to -w- (new file), to change that behaviour.

File handler has alogging level of DEBUG, so everything is logged. However, console handler logging level is controlled by the SAS Verbosity (SAS_VERBOSITY env. variable) which is set via the option -V.

The logger can not be called more than once. If not, messages start appearing duplicated. That is why the logging method is separarted from the logger definition.

log(level, msg)[source]

A log message will be issued according to the logging level. The logging level is fixed for the log file but on the console will depend on SAS_VERBOSITY. SAS_VERBOSITY can be set between 1 (min) and 10 (max). Correspondence between SAS_VERBOSITY levels and logging levels

DEBUG -> verbosity = 8, 9, 10

INFO -> verbosity = 6, 7

WARNING -> verbosity = 4, 5

ERROR -> verbosity = 2, 3

CRITICAL -> verbosity = 1

Read Parameter Files

Defines class paramXmlInfoReader with methods to parse and extract all the information included in the task parameter file and build several data structures which allow to handle them when entered either from the command line or via an argument dict.

class pysas.param.paramXmlInfoReader(taskname, logger=None)[source]

Bases: object

Class which provides several methods to read and handle all the information contained in the the task parameter file. The file is formatted according to the DOM standard. Once the information is loaded, it is distributed into several data structures to allow proper handling of any parameters entered from the command line or through an argument list.

xmlParser()[source]

Parses the parameter file into a Document object.

The first task of xmlParser is to get the whole XML file as an object of type Document (For a list of all objects, look at the xml.dom documentation). Then, it produces a list of Element objects in the paramater file which are identified by the tag ‘PARAM’. This is done by means of the Element method getElementsByTagName.

Outputs:

  • params : List of all elements with tag PARAM

  • allparams : Dictionary { parameter: {attributes} }

  • pels : Dictionary { element: pname }

  • rev_pels : Dictionary { pname : element }

  • parmap : Dictionary { pname: [ subparameters …], …}

It is the map of the parameter tree as a list of dictionaries. Each dictionary corresponds to a paramater and all its sub-parameters structure as follows:

keys are the parent parameters and values are lists with all their subparameters

Examples:

1- Parent parameter: keepfilteroutput

Subparameters: withfilteredset, filteredset

{‘keepfilteroutput’: [‘withfilteredset, ‘filteredset’], ‘wilthfilteredset’: [‘filteredset’], ‘fileteredset’: [] }

This dictionary corresponds to a parameter tree as follows:

keepfilterouput
└── withfilteredset

└── filteredset

2- Parent parameter: filtertype

Subparameters: dssblock, expression

{‘filtertype’: [‘dssblock’, ‘expression’], ‘dssblok’: [], ‘expression’: [] }

filtertype

├── dssblock

└── expression

3- Parent parameter: table

Subparameters: None

{‘table’: [] }

  • mandpar : List of all parameters/subparameters defined mandatory = ‘yes’.

  • mainparams : List of main paramaters, with or without subparamaters

  • mandpar_dict : Dictionary, key = mandatory subparamater, value=its parent parameter

  • rev_mandpar_dictDictionary, keys = parent paramater,

    value = list of all mandatory subparameters

  • rev_mandpar_string_dictDictionary, keys = parent parameter type ‘string’

    value = list of alternatives

printHelp()[source]

Prints a table with the parameters read from xmlFile.

defaultValues()[source]

Returns a dictionary with the default values for each parameter.

Default values might be of different types with multiple values. For the time being, we create lists for all of then, which can be either single or multiple values.

Returns:

Dictionary of the default values for the SAS task.

Return type:

dict

static att(p)[source]

Obtains the attributes of parameter p from the parameter file.

Parameters:

p (NodeList) – XML NodeList from xml.dom.

Returns:

Dictionary with the parameter attributes.

Return type:

dict

static getsub(p)[source]

Returns the list of all subparamaters from a given parameter p downwards the tree structure.

Parameters:

p (NodeList) – XML NodeList from xml.dom.

Returns:

Returns the NodeList for the subparameters for parameter p.

Return type:

NodeList

static nam2el(pname, rev_pels)[source]

Returns the parameter element p for a parameter name.

Parameters:

  • pname (str) – Parameter name.

  • rev_pels (dict) – Dictionary of NodeLists for the parameters.

Returns:

The NodeList for the parameter name.

Return type:

NodeList

Raises:

Exception – rev_pels is undefined.

static el2nam(p, pels)[source]

Returns the name of a parameter element p.

Parameters:

  • p (NodeList) – NodeList of parameter to find.

  • pels (dict) – Dictionary of parameter names based on parameter elements.

Returns:

_description_

Return type:

_type_

Raises:

Exception – pels is undefined.

class pysas.param.SASParams(*args, **kwargs)[source]

Bases: UserDict

The class ‘SASParams’ is a child of ‘UserDict’, which is like a normal Python dict, but allows for special modifications.

Upon instatiation the object will create a parallel dict which keeps track if the values of the main dict have been modified.

The end user should interact with the SASParams dict like a normal Python dictionary.

How to use:

from pysas.param import SASParams

my_task_inputs = SASParams({})

my_task_inputs.set_inparams_to_defaults(taskname)

set_inparams_to_defaults(taskname)[source]

Sets special dictionary ‘inparams’ to defaults without setting ‘modified’ to ‘True’. Silently overwrites previous values!

Parameters:

taskname (str) – SAS task name.

pysas.param.get_input_params(taskname)[source]

Function to return SASParams dictionary of input parameters for a given SAS task.

Parameters:

taskname (str) – SAS task name.

Returns:

SASParams dictionary. Special dictionary of input parameters with defaults. Behaves like a normal dictionary, but keeps track of which parameters have been modified.

Return type:

SASParams

Parse Inputs

The parser module implements the class ParseArgs which includes all the methods required to parse any arguments entered either via the command line or through MyTask.

The class initialization checks for the existence of SAS_PATH. Then it checks the version of the SAS task.

This module is for internal use. It is not intended for the end user to use this module.

class pysas.parser.ParseArgs(taskname, argdict, logger=None)[source]

Bases: object

Class for parsing the input arguments. For pySAS v2.0 it is now assumed that the input arguments are passed in as a dictionary.

optparser()[source]

This parses the ‘options’ from the input. The parsed options are stored in a dict called parsedargs.

There are two groups of options:

  1. Immediate action:

    If the option is present, it must be processed immediately and exit. These are:

    -v/–version

    -d/–dialog

    -m/–manpage

    -h/–help

    -p/–param

  2. Modifiers:

    If the option is present, it modifies the execution of the command and/or the environment, usually by setting an environment variable. These are:

    -V/–verbosity (SAS_VERBOSITY)

    -c/–noclobber (SAS_CLOBBER)

    -a/–ccfpath (SAS_CCFPATH)

    -i/–ccf (SAS_CCF)

    -o/–odf (SAS_ODF)

    -f/–ccffiles

    -w/–warning

    -t/–trace.

exe_options()[source]

Method to execute options or parameters that require immediate action.

These include options that execute the command and exit. These are:

version(–version, -v)

help(–help, -h)

param(–param, -p)

dialog(–dialog, -d)

manpage(–manpage, -m)

Returns ‘Exit’ which if True will send the exit command up the chain. If False, then pySAS will continue to execute.

Returns:

Whether to exit.

Return type:

bool

env_options()[source]

Method to collect options that set environment variables and then continue running the SAS task. These are:

-V/–verbosity (SAS_VERBOSITY)

-c/–noclobber (SAS_CLOBBER)

-a/–ccfpath (SAS_CCFPATH)

-i/–ccf (SAS_CCF)

-o/–odf (SAS_ODF)

-f/–ccffiles

-w/–warning

-t/–trace

Returns:

return_options, a dictionary to hold set options.

Return type:

dict

SAS Utilities

Utility functions specific to SAS or pySAS.

pysas.sasutils.download_data(obsid, data_dir, level='ODF', repo='esa', overwrite=True, logger=None, encryption_key=None, proprietary=False, credentials_file=None, PPS_subset=False, instname=None, expflag=None, expno=None, product_type=None, datasubsetno=None, sourceno=None, extension=None, filename=None, **kwargs)[source]

Not intended to be used by the end user. Internal use only. Use obsid.download_ODF_data(), obsid.download_PPS_data(), or obsid.download_ALL_data() instead.

Downloads, or copies, data from chosen repository.

Will silently overwrite any preexisting data files and remove any existing pipeline products. Will create directory structure in ‘data_dir’ for odf.

Only the download_data functions in ObsID will check for preexisting files.

Parameters:

  • obsid (str) – Obs ID in string format.

  • data_dir (str) – Path to directory where the data will be downloaded. Automatically creates directory data_dir/obsid. data_dir MUST exist.

  • level (str, optional) – Level of data products to download. Allowed values are ‘ODF’, ‘PPS’, or ‘ALL’, by default ‘ODF’.

  • repo (str, optional) – Which repository to use to download data. Not case sensitive. Allowed values are ‘esa’,’xsa’,’heasarc’,’nasa’,’sciserver’,’fornax’, or ‘aws’, by default ‘esa’.

  • overwrite (bool, optional) – Will only check if obs_dir exists, by default True.

  • logger (logger, optional) – logger object, by default None

  • encryption_key (str, optional) – Encryption key for proprietary data, a string 32 characters long. -OR- path to file containing ONLY the encryption key, by default None.

  • proprietary (bool, optional) – Flag for downloading proprietary data from the XSA at ESA, by default False.

  • credentials_file (str, optional) – Path and filename of file containing XSA username and password. For proprietary data only. (If not given then astroquery will ask user for username and password.) Defaults to None.

  • PPS_subset (bool, optional) – Set PPS_subset=True if downloading a subset of PPS. Defaults to False.

  • instname (str, optional) – Instrument name. Defaults to None.

  • expflag (str, optional) – Exposure flag. Defaults to None.

  • expno (int or str, optional) – Exposure number. Defaults to None.

  • product_type (str, optional) – PPS product type. Defaults to None.

  • datasubsetno (str, optional) – Data subset number/character. Defaults to None.

  • sourceno (int or str, optional) – Source number or slew step number. Defaults to None.

  • extension (str, optional) – File format/extension. Defaults to None.

  • filename (str, optional) – If the exact PPS file name is known (no wildcards), then this can be used to download a single PPS file. Defaults to None.

  • **kwargs – Additional keyword arguments passed through to underlying download handler (Astroquery).

Raises:

  • Exception – File {odftar} extension not recognized.

  • Exception – tar file extraction failed.

  • Exception – Obs ID {obsid} not found in the XMM archive!

  • Exception – File download failed!

  • Exception – Multiple possible encryption key files.

  • Exception – File decryption failed. No encryption file found.

  • Exception – File decryption failed. No encryption key found.

  • Exception – File decryption failed.

pysas.sasutils.generate_logger(logname='general_sas', log_dir=None)[source]

Function to generate a Task Logger. Note: For the _old_ task logger.

Parameters:

  • logname (str, optional) – Filename for log file, by default general_sas.

  • log_dir (str, optional) – Directory for the log file, by default current directory or environment variable ‘SAS_TASKLOGDIR’.

Returns:

File logger object.

Return type:

logger

pysas.sasutils.update_calibration_files(repo='NASA')[source]

Function to download/update XMM calibration files.

Parameters:

repo (str, optional) – Download repository. Allowed options are ‘NASA’ and ‘ESA’. Not case sensitive. By default ‘NASA’.

Returns:

Subprocess return.

Return type:

result

Raises:

Exception – SAS_CCFPATH not set.

pysas.sasutils.install_sas(repo='NASA', sas_version='21.0.0')[source]

!!WARNING!! EXPERIMENTAL! No guarentees this will work. Only tested with Ubuntu v. 20 and 22

Parameters:

  • repo (str, optional) – Download repository. Allowed options are ‘NASA’ and ‘ESA’. Case sensitive. By default ‘NASA’.

  • sas_version (str, optional) – SAS version to download. By default ‘21.0.0’.

Raises:

  • Exception – Repository ‘{repo}’ not recognized.

  • Exception – Ubuntu version {version} not recognized.

  • Exception – Linux distribution {distribution} not supported.

pysas.sasutils.generate_PPS_pattern(obsid=None, instname=None, expflag=None, expno=None, product_type=None, datasubsetno=None, sourceno=None, extension=None)[source]

Function for generating a filename for downloading PPS files.

PPS data product filenames take the 27.3 character form:

POOOOOOOOOODDUEEETTTTTTSXXX.FFF

P

The character P, to identify the files as a PPS product file

OOOOOOOOOO

(obsid) Observation identifier

DD

(instname) Data source identifier (instrument name)

U

(expflag) Exposure flag (1 character = S (sched), U (unsched), X (not applicable))

EEE

(expno) Exposure number within the instrument observation (3 digits)

TTTTTT

(product_type) Product type (6 characters)

S

(datasubsetno) 0 or data subset number/character (1 character, differentiates energy bands, OSWs, filters, orders etc.)

XXX

(sourceno) Source number or slew step number (3 characters, hexadecimal). It is set to 000 in source products from EPIC-pn Timing mode.

FFF

(extension) File format (3 characters)

If inputs are not given then a wildcard “*” character will be inserted.

Parameters:

  • obsid (str, optional) – Obs ID, by default None.

  • instname (str, optional) – Data source identifier (instrument name), by default None.

  • expflag (str, optional) – Exposure flag, by default None.

  • expno (str, optional) – Exposure number, by default None.

  • product_type (str, optional) – Product type, by default None.

  • datasubsetno (str, optional) – Data subset number/character, by default None.

  • sourceno (str, optional) – Source number or slew step number, by default None.

  • extension (str, optional) – File format, by default None.

Returns:

PPS filename pattern with wildcards.

Return type:

str

pysas.sasutils.load_json_from_package(filename)[source]

Load a JSON file from a given package using importlib.resources.files().

Parameters:

filename (str) – The JSON file name inside the package.

Returns:

Parsed JSON data.

Return type:

dict

pySAS and SAS Version

pysas.version.get_sas_version()[source]

Function to return information about the SAS version. Runs ‘sasversion’ and parses the output.

Returns:

Dictionary of SAS version information.

Return type:

dict

pysas.version.print_sas_version()[source]

Shows release and information about the XMM-Newton Science Analysis System (SAS).

Raises:

EnvironmentError – SAS_DIR is undefined.