Giter VIP home page Giter VIP logo

chromos's Introduction

Chromos

Chromos is a software pipeline written to automate the extraction and analysis of RXTE data, allowing a variety of spectral-timing properties to be determined for a large number of observations.

Please note that the bulk of this work was done between July 2015-2016 as part of a MSc project, and as such is not intend to be maintained.

Installation

Software can be installed by cloning the Chromos repository, and may need additional supporting software if not present on the user's system. Chromos has been designed for use on Taurus, running Ubuntu 14.04.4. A list of required software is documented in REQUIREMENTS.txt, detailing the recommended version of particular software packages. To ensure the full pipeline works on Taurus, a virtual environment will need to be installed. However, those not doing energy spectral analysis can run Chromos without requiring a virtual environment with additional packages.

Using Chromos

Chromos has been built in modular fashion, allowing for the quick toggling of required components. An overview of some of the current steps in Chromos is presented below, showing the order in which the pipeline must be run with main_pipeline.py. Parameters for running this file are set in paths.py, which can be modified to reflect where data should be stored, under which name etc. The pipeline can be run over multiple sources using scripts in the misc folder, being quick hacks to change the paths file before running the main pipeline. As running Chromos on a full set of observations for a single source can take several hours, full Chromos runs are best run with screen, a command allowing scripts to continue to be run while disconnected from the terminal. The only required information apart from that presented in paths.py is a list of ObsIDs over which you wish to run Chromos. A list of ObsIDs can be found for each object using the HEASARC archive: http://heasarc.gsfc.nasa.gov/cgi-bin/W3Browse/w3browse.pl

A choice can be made to run Chromos for extracting data, calculating power spectra, all the way up to calculating power colours. Additionally energy spectral analysis can also be done, if wished, skipping the whole timing analysis side of Chromos. Currently Chromos works for event, binned, goodxenon and std2 files, extracting all types to allow the choice of data mode to be determined only when necessary.

Modules

The following paragraphs briefly explain the purpose of each step in Chromos, providing a way to choose which components are necessary in your own pipeline. A log of the latest run of each step is automatically saved, allowing for future reference. The abbreviation TA refers to timing analysis scripts, which aren't required to run energy spectral analysis, denoted with SA, and vice-versa.

Download Data Based primarily on scripts coded by Abbie Stevens, this script checks whether you already have a copy of the required ObsIDs, and if not, will download a copy from the NASA archives to the location specified in paths.py

Locate Files Using Phil Uttley's xtescan2, an overview of data files is created in each 'P-folder', the parent folders to the ObsID directories. This overview contains information on all locations, data types, observation dates etc, for each data file in the folder.

Determine Info A crucial step in Chromos, this code runs through all of the files created in the previous step, creates a database with separate entries of each data file and saves it to a csv file as given in paths.py. Despite occasional I/O problems regarding the speed of csv files, the decision for this data storage was made to ensure portability and user-friendliness. All subsequent steps require a database file, and will update it upon completion. If wishing to rerun Chromos in its entirety from this step, the database file is best deleted manually to prevent a contamination between previous runs and a new run.

Spacecraft Filters Following standard steps in the RXTE cookbook, GTI files are made to filter erroneous data such as when the observations are blocked by the Earth.

GoodXenon to FITS A necessary step in extracting GoodXenon data requires the original data files, consisting of two parts, to be matched together. Ideally this step would still take place, but on a different level, allowing Chromos to deal with goodxenon files in the same manner as other data modes.

PCU Filters To prevent a surge in current from contaminating data, a cut is made around the time at which the number of PCU changes.

Create Backgrounds Contamination from off-sources photons can be modelled with a background. This step creates background data files which are modelled of basis of the settings of the actual data files. Backgrounds can only be extracted at a maximum time resolution of 16s, and therefore require additional steps after extraction, and before using for analysis.

Find Channels Perhaps the most intricate part of Chromos, this script finds the channels required for the specified energy range. Determining these channels for each data file can currently only be done for one energy range at a time, but can easily be adapted.

Extract LC and SP The most time-consuming part of Chromos, requiring all output of the previous steps to be gathered for input here. Light curves are extracted for all data modes save for std2 files, for which both light curves and spectra are extracted. The latter are only extracted for PCU2, as it performed the most consistently over the course of the full RXTE mission. Currently all data is extracted at the same resolution, however support has been in built to allow for different resolutions per data file, with subsequent files still defined by data mode and maximum possible resolution.

Correct for background (TA) As background files are only extracted at a 16s time resolution, this script interpolates between values to obtain background rates at the same resolution as the required light curve. This is subtracted from the light curve, and saved to a new file for subsequent steps.

Create Power Spectra (TA) Another time-intensive step, this calculates a power spectrum for each observation, which is split up into multiple parts of a predefined length. As an essential step in calculating power colours, this code has been extensively commented.

Create Power Colours (TA) The final step in the timing analysis -- calculating power colours for as many power spectra as possible. Currently no simple way exists for extracting a simple file with ObsIDs and the corresponding power colours, as this currently requires filtering of the database. Scripts with these filters can be found in the misc folder, allowing power colours to be selected upon 3sigma constraints, timing resolution or otherwise.

Create Responses (SA) Script allowing response files to be generated for each spectrum, ready for input into xspec.

Calculate HI (SA) Based on Fortran scripts developed by Phil Uttley, this code calculates the hardness and intensity for in predefined energy bands, saving the results, like every other step, to the database.

Accessing Data

All data calculated by Chromos can be found in the database, apart from in cases when size prohibited inclusion, in which case the path to the file is noted. Logs are also created by Chromos allowing individual files to be traced back through processing. The sheer size of the database leads to the pandas package being the recommended package with which to interact with the database. The amount of detail in each database requires substantial filtering on various parameters before running scripts over columns. Investigating the manner in which previous columns were created will be essential if you wish to filter this data yourself. Alternatively, self-made scripts could search for the various files created with Chromos, which should have recognizable and distinct names.

Notes

Please note that the bulk of this work was done between July 2015-2016 as part of a MSc project, and as such is not intend to be maintained.

As a brief side note for those who are curious: the name Chromos is an amalgamation of Chroma (GR: colour) and Chronos (GR: god of time). Referring respectively to the power colour analysis Chromos is capable of, and the timing capabilities of RXTE, the name is designed to evoke the idea of time and evolution, and form a sense of reliability.

Improvements

Some large-scale improvements which could be made to Chromos are the manner in which it deals with GoodXenon files, which could be substantially simplified, but also an rewrite to adopt a class structure. This would help allow Chromos to be more adaptable, giving more freedom for a wider range of extraction parameters. Scripts should also be written to filter the data created by Chromos to a simple data files with only the necessary parameters, allowing users to avoid the intricacies of data filtering. A lot of this functionality has already been developed for scripts in the misc folder, or scripts in the plots folder. The latter directory is perhaps the best place to start with learning how to obtain the required data from a database file, and will help in checking the data by using pre-coded plotting scripts.

Author

David Gardenier | MSc student at the University of Amsterdam

chromos's People

Contributors

astrocoding avatar davidgardenier avatar dependabot[bot] avatar

Stargazers

 avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

chromos's Issues

Indents should be spaces!

FYI: indents should be spaces, not tabs. Four spaces is standard for one indent level.

Most (fancier) text editors will let you specify that hitting the 'tab' key actually uses 4 spaces, so that you can indent with spaces and still be lazy (by only hitting one key).

Extraction of event 62us files fails

Aborts extraction as "Filtering resulted in having no data!". Could perhaps have to do with the bin size (usually 1/128s) that you give as input?

Allow for downloading of <obsids>G folders

At the moment I don't download obsids folders with a letter behind them, but in theory folders with a 'G' behind it should work (means _G_appy, so has longer time gaps in the observations).

Splitting out standard2 files

The output of Phil's xtescan2 provides Standard2 lists, linking to original files. However, the list can change if you run it multiple times, in later runs including .bkg and .std2bkg files. I've quickly dealt with this by taking every third line - but it is only valid if you've run the pipeline multiple times. If you were to run the pipeline only once (over a new data set), you would need every line. The root of the problem lies in finding why and how those 'extra' files are being created. Only related to pc_ss from what I can tell.

Correct for the SAA

Simple task of adding a before and after Southern Atlantic Anomaly filter in the maketime filter creation.

Length of Obsid

One file (20092-01-05-030) has a obsid which is one character longer than all other obsids. Unsure whether any part of the code may trip up due to this.

Creating responses for observations which straddle epochs

At the moment if an observation falls exactly on the boundary between two epochs, pcarsp won't create a response file, but asks you to rerun it with the '-d' option to shift the observation by a day. I skip these files at the moment, but these should ideally be caught and rerun with the '-d' option.

X-ray flares across multiple modes

It seems that sometimes X-ray flares are detected in say event mode files, but not in the binned mode files. This could be because the binned mode files have a different time range (and are usually only in the order of seconds), but could also be because the count rate never gets above 7sigma. Ideally I should check which case is happening, but at the moment none of the binned files pass the power spectra step anyway, so not too concerned.

Runtime calculation

Runtime calculation in logging currently shows full runtime rather than runtime per subscript.

Return 512 time ranges

If wanting to group power colour ratios in sets other than the arbitrary time ranges of an obsid, you'll need power colour ratios per time units (in this case per 512s would be a good start - you'll not want to go to a lower time-resolution).

Response files with ampersand

Response files are occasionally being read in to XSPEC with an ampersand at the end of the file name. I thought I had fixed this earlier in the year, but still occasionally crops up. Perhaps the version of astropy/xspec have changed in the mean time, leading to different behaviour. The quick fix is to (manually) copy the response name (without including the ampersand) into the queried input on which XSPEC stops, and press enter. Means some additional oversight is needed for some objects (unsure when this issue does or doesn't pop up).

Background creation in pc_ss

Relates to master_project/pc_ss/D_create_background.py, fixed in pc_hid


If using pcabackest to create backgrounds for std2 files - then fullspec should equal 'no'. Although if using pcabackest for other files which also need no gaincorrection, it should be 'yes'.

Cut around PCU changes

Currently the filter files do not cut on basis of PCU changes. Might not even have to implement it, but perhaps plotting will be enough to show PCU changes have taken place. If cutting, remove 32 seconds around the change.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.