JOSS Python36 Licence

dfitspy as an executable

Note

All the files used in the examples below are public and accessible via the ESO archive.

1-The command line interface help

You can start dfitspy from a terminal. dfitspy comes with a command line interface which includes a ‘help’ that you can display in your terminal using the help command. It must be called like this:

[user@machine]$ dfitspy --help

This command will display the help of the program:

usage: dfitspy [-h] [-file [FILE [FILE ...]]] [-key KEY] [--list] [--grep GREP] [--exact] [--HDU HDU]
               [--save] [--test] [--version] [--docs]

dfitspy: dfits|fitsort in python, version 20.7.0, Licence: GPL

optional arguments:
  -h, --help            show this help message and exit

Mandatory arguments if you want to dfitsort your files:
  -file [FILE [FILE ...]]
                        a file, a list of file separated by coma, *.fits is accepted, * as well, test* as
                        well, testdir/test* as well
  -key KEY              Header keyword or list of header keywords (separated by coma)

Optional arguments:
  --list                List all keywords in a given file (if a list of file is given the first one is
                        used)
  --grep GREP           Restrain the files to the one with a given value of a given parameter. It can be
                        used multiple times with different values
  --exact, -e           Consider only exact matches for keywords
  --HDU HDU, -H HDU     HDU extension, default is 0 (primary header)
  --save                Save the list of files into an ascii file
  --test                Start the testing of the program
  --version             Display the version of the program
  --docs                Diplay the online or local documentation program

In details it means:

dfitspy has few optionnal arguments and 2 semi mandatory arguements. You can not start dfitspy without any argument:

  • -h and ‘- -help’: Display this help in the terminal.

2-Giving files

The argument ‘-file’ [or the shorter version ‘-f’] is where you feed dfitspy with your files. You have multiple option how to give values to this argument:

  • A single file: -file test.fits

  • Multiple particular files: -file test.fits,test2.fits. They must be separated with comas and without spaces

  • You can use the ‘all’ argument: -file all

  • You can use the ‘*’ solution: -file *

  • Semi complete names: -file test* or -file *test.fits or with path: -file /test/*test.fits.

Warning

The directory that is considered here is the current directory where dfitspy is started.

3-Giving keywords

The argument ‘-key’ [or the shorter version ‘-k’] is where you precise what keywords dfitspy must retrieve in the headers. Here again you have multiple option

  • a single keyword: -key OBJECT

  • multiple arguments: -key OBJECT,LST. They must be separated by comas and without spaces.

Warning

Two things you must be careful of:

  • If your keywords contain spaces, ex: HIERARCH ESO OBS ID, you must replace it by dots: HIERARCH.ESO.OBS.ID

An example of output is the following (running the command: dfitspy -f * -k HIERARCH.ESO.OBS.ID,EXPTIME):

[Command: dfitspy -f * -k ESO.OBS.ID,EXPTIME]

[DFITSPY INFO]> Current directory: /home/romain/Documents/tests/dfitspy
[DFITSPY INFO]> 24 fits files will be considered
[DFITSPY INFO]> We look in HDU 0

              filename                  ESO OBS ID      EXPTIME
-------------------------------------   ----------      -------
XSHOOTER.2014-06-19T05:00:28.009.fits    1072941         300.0
XSHOOTER.2014-06-19T05:00:53.001.fits    1072941         250.0
XSHOOTER.2014-06-19T05:01:40.463.fits    1072941         150.0
XSHOOTER.2014-06-19T05:10:40.714.fits    1072971         10.0
XSHOOTER.2014-06-19T05:10:41.562.fits    1072971          5.0
XSHOOTER.2014-06-19T05:10:41.934.fits    1072971          1.5
XSHOOTER.2014-06-19T05:11:15.667.fits    1072971         10.0
XSHOOTER.2014-06-19T05:11:16.622.fits    1072971          5.0
XSHOOTER.2014-06-19T05:11:16.887.fits    1072971          1.5
XSHOOTER.2014-06-19T05:12:56.978.fits    1072971          0.0
XSHOOTER.2014-06-19T05:13:33.009.fits    1072971         300.0
XSHOOTER.2014-06-19T05:13:58.001.fits    1072971         250.0
XSHOOTER.2014-06-19T05:14:45.502.fits    1072971         150.0
XSHOOTER.2014-06-19T05:24:45.749.fits    1072985         10.0
XSHOOTER.2014-06-19T05:24:46.590.fits    1072985          5.0
XSHOOTER.2014-06-19T05:24:46.969.fits    1072985          1.5
XSHOOTER.2014-06-19T05:25:18.621.fits    1072985         10.0
XSHOOTER.2014-06-19T05:25:19.503.fits    1072985          5.0
XSHOOTER.2014-06-19T05:25:19.841.fits    1072985          1.5
XSHOOTER.2014-06-19T05:26:29.578.fits    1072985          0.0
XSHOOTER.2014-06-19T05:27:07.001.fits    1072985         300.0
XSHOOTER.2014-06-19T05:27:32.003.fits    1072985         250.0
XSHOOTER.2014-06-19T05:28:20.452.fits    1072985         150.0
XSHOOTER.2014-06-19T05:40:13.194.fits    1073005         150.0

[DFITSPY INFO]> 24 files used in output

First dfitspy reminds you what directory we are in, then it tells you the number of files that were considered with the command, this number depends on what you give through the ‘-f’ argument, and finally it will gives what is the HDU we are looinkg for. Then it displays the header always starting with the filename and then all the keywords the user requires to be displayed. And It displays the list of all the filenames and fits header values. Finally, it gives you the number of files in the list.

It is important to understand the following things to fully understand the output of dfitspy:

  • By default, dfitspy will look for all the header keywords that contain the one you gave with the ‘-k’ argument. Using the file in the previous example, if I want the keyword NAXIS, dfitspy will retrieve ALL the following keywords: NAXIS, NAXIS1 and NAXIS2:

    [Command: dfitspy -f * -k ESO.OBS.ID,NAXIS]
    
    [DFITSPY INFO]> Current directory: /home/romain/Documents/tests/dfitspy
    [DFITSPY INFO]> 24 fits files will be considered
    [DFITSPY INFO]> We look in HDU 0
    
                  filename                  ESO OBS ID      NAXIS   NAXIS1  NAXIS2
    -------------------------------------   ----------      -----   ------  ------
    XSHOOTER.2014-06-19T05:00:28.009.fits    1072941          2      2144    3000
    XSHOOTER.2014-06-19T05:00:53.001.fits    1072941          2      2106    4000
    XSHOOTER.2014-06-19T05:01:40.463.fits    1072941          2      2048    1100
    XSHOOTER.2014-06-19T05:10:40.714.fits    1072971          2      1000    1000
    XSHOOTER.2014-06-19T05:10:41.562.fits    1072971          2      2048    1100
    XSHOOTER.2014-06-19T05:10:41.934.fits    1072971          2      1000    1000
    XSHOOTER.2014-06-19T05:11:15.667.fits    1072971          2      1000    1000
    XSHOOTER.2014-06-19T05:11:16.622.fits    1072971          2      2048    1100
    XSHOOTER.2014-06-19T05:11:16.887.fits    1072971          2      1000    1000
    XSHOOTER.2014-06-19T05:14:45.502.fits    1072971          2      2048    1100
    XSHOOTER.2014-06-19T05:40:13.194.fits    1073005          2      2048    1100
    .
    .
    .[This has been cut]
    

This has been implemented so you can ‘search’ for keywords without knowing their exact names. If you happen to know the exact name of the keyword and you don’t want the other ones, you can use the ‘–exact’ argument [or the shorter version ‘-e’] and you will have only the keywords matching exactly the ones you asked for:

[Command: dfitspy -f * -k ESO.OBS.ID,NAXIS -e]

[DFITSPY INFO]> Current directory: /home/romain/Documents/tests/dfitspy
[DFITSPY INFO]> 24 fits files will be considered
[DFITSPY INFO]> We look in HDU 0

      filename                          ESO OBS ID      NAXIS
-------------------------------------   ----------      -----
XSHOOTER.2014-06-19T05:00:28.009.fits    1072941          2
XSHOOTER.2014-06-19T05:00:53.001.fits    1072941          2
XSHOOTER.2014-06-19T05:01:40.463.fits    1072941          2
XSHOOTER.2014-06-19T05:10:40.714.fits    1072971          2
XSHOOTER.2014-06-19T05:10:41.562.fits    1072971          2
XSHOOTER.2014-06-19T05:13:58.001.fits    1072971          2
XSHOOTER.2014-06-19T05:14:45.502.fits    1072971          2
.
.
.
.[This has been cut]
  • In case a file does not contain the keyword you are requesting, its value in the table will be replaced by ‘-‘:

    [Command: dfitspy -f * -k ESO.OBS.ID,OCS.ARM]
    
    [DFITSPY INFO]> Current directory: /home/romain/Documents/tests/dfitspy
    [DFITSPY INFO]> 24 fits files will be considered
    [DFITSPY INFO]> We look in HDU 0
    
                  filename                  ESO OBS ID      ESO OCS ARM
    -------------------------------------   ----------      -----------
    XSHOOTER.2014-06-19T05:00:28.009.fits    1072941             -
    XSHOOTER.2014-06-19T05:00:53.001.fits    1072941             -
    XSHOOTER.2014-06-19T05:01:40.463.fits    1072941             -
    XSHOOTER.2014-06-19T05:10:40.714.fits    1072971             -
    XSHOOTER.2014-06-19T05:11:16.622.fits    1072971             -
    XSHOOTER.2014-06-19T05:11:16.887.fits    1072971             -
    XSHOOTER.2014-06-19T05:12:56.978.fits    1072971           AGCCD
    XSHOOTER.2014-06-19T05:13:33.009.fits    1072971             -
    .
    .
    .[This has been cut]
    
  • HIERARCH keywords: ESO FITS files come with huge collection of keywords starting by the HIERARCH prefix. You can ask for them using the HIERARCH prefix or without it:

    [Command: dfitspy -f * -k HIERARCH.ESO.OBS.ID,NAXIS -e]
    
    [DFITSPY INFO]> Current directory: /home/romain/Documents/tests/dfitspy
    [DFITSPY INFO]> 24 fits files will be considered
    [DFITSPY INFO]> We look in HDU 0
    
                  filename                  HIERARCH ESO OBS ID     NAXIS
    -------------------------------------   -------------------     -----
    XSHOOTER.2014-06-19T05:00:28.009.fits         1072941             2
    XSHOOTER.2014-06-19T05:00:53.001.fits         1072941             2
    XSHOOTER.2014-06-19T05:01:40.463.fits         1072941             2
    XSHOOTER.2014-06-19T05:10:40.714.fits         1072971             2
    XSHOOTER.2014-06-19T05:10:41.562.fits         1072971             2
    XSHOOTER.2014-06-19T05:10:41.934.fits         1072971             2
    XSHOOTER.2014-06-19T05:11:15.667.fits         1072971             2
    .
    .
    .[This has been cut]
    

You will have the same result by issuing the following command: dfitspy -f * -k ESO.OBS.ID,NAXIS -e

4-GREPPING option

dfitspy offers you the possibility to display files with particular values that you are expecting. This is done using the option - - grep (double dash) and giving as argument an expected value of a keyword given with the -key option. Doing so will tell dfitspy to consider the files only if one of the keyword that the user asks to display has the grepping value.

Simple grepping

Taking the same command as above, one might want to get only the files with ESO.OBS.ID = 1072985. This is easily assing the following flag: - -grep 1072985 [double dash without space]. This will produce the following output in terminal:

[command: dfitspy -f * -k ESO.OBS.ID,NAXIS,SEQ.ARM --grep 1072985]

[DFITSPY INFO]> Current directory: /home/romain/Documents/tests/dfitspy
[DFITSPY INFO]> 24 fits files will be considered
[DFITSPY INFO]> We look in HDU 0

              filename                  ESO OBS ID      ESO SEQ ARM     NAXIS   NAXIS1  NAXIS2
-------------------------------------   ----------      -----------     -----   ------  ------
XSHOOTER.2014-06-19T05:24:45.749.fits    1072985            UVB           2      1000    1000
XSHOOTER.2014-06-19T05:24:46.590.fits    1072985            NIR           2      2048    1100
XSHOOTER.2014-06-19T05:24:46.969.fits    1072985            VIS           2      1000    1000
XSHOOTER.2014-06-19T05:25:18.621.fits    1072985            UVB           2      1000    1000
XSHOOTER.2014-06-19T05:25:19.503.fits    1072985            NIR           2      2048    1100
XSHOOTER.2014-06-19T05:25:19.841.fits    1072985            VIS           2      1000    1000
XSHOOTER.2014-06-19T05:26:29.578.fits    1072985            AGC           2      562     528
XSHOOTER.2014-06-19T05:27:07.001.fits    1072985            UVB           2      2144    3000
XSHOOTER.2014-06-19T05:27:32.003.fits    1072985            VIS           2      2106    4000
XSHOOTER.2014-06-19T05:28:20.452.fits    1072985            NIR           2      2048    1100

[DFITSPY INFO]> 10 files used in output

Multi grepping

We can add this flag multiple time. In the previous command, if we want only the files for the NIR arm we must add another grepping flag:

[command: dfitspy -f * -k HIERARCH.ESO.OBS.ID,NAXIS,SEQ.ARM --grep 1072985 --grep NIR]

[DFITSPY INFO]> Current directory: /home/romain/Documents/tests/dfitspy
[DFITSPY INFO]> 24 fits files will be considered
[DFITSPY INFO]> We look in HDU 0

              filename                  ESO SEQ ARM     HIERARCH ESO OBS ID     NAXIS   NAXIS1  NAXIS2
-------------------------------------   -----------     -------------------     -----   ------  ------
XSHOOTER.2014-06-19T05:24:46.590.fits       NIR               1072985             2      2048    1100
XSHOOTER.2014-06-19T05:25:19.503.fits       NIR               1072985             2      2048    1100
XSHOOTER.2014-06-19T05:28:20.452.fits       NIR               1072985             2      2048    1100

[DFITSPY INFO]> 3 files used in output

5-Files with multi extension headers

The ‘–HUD’ arguement [or the shorter -H] allows you to explore headers of different extensions. dfitspy will read only the header of the extension given by –HDU, and this for all files. By default, the primary header (HDU=0) is used. An example with the header of the first extension:

[Command dfitspy -f tfn0m410-kb98-2100-x00.fits.fz -H 1 -k POLARMOX,SUNALT,MOONDIST]

[DFITSPY INFO]> Current directory: /home/romain/Documents/tests/lcogtdata-20200722-1
[DFITSPY INFO]> 1 fits files will be considered
[DFITSPY INFO]> We look in HDU 1

               filename                  MOONDIST       POLARMOX          SUNALT
--------------------------------------- ----------      --------        -----------
tfn0m410-kb98-2100-x00.fits.fz          99.9823565       0.1979         -23.9730277

[DFITSPY INFO]> 1 files used in output

If a file does not have this extension, dfistpy will report an error and quit. The error looks like: ‘ValueError: HDU 1 not in XSHOOTER.2014-06-19T05:00:28.009.fits’

6-Extra arguments

Few extra arguments can be used:

  • ‘- -test’: This runs the tests (written with unittest library) of dfitspy.

  • ‘- -list’: This must be used with the -file option described above. It takes the first file given by ‘-file’ and display a list of all the arguments in a 3 columns fashion. Example:

    [DFITSPY INFO]>keywords in r.XSHOO.2099-14-59T09:53:43.577_tpl-A01_0000.fits
    --------------------------------------------------------------------------------
    SIMPLE                         | BITPIX                         | NAXIS
    EXTEND                         | COMMENT                        | DATE
    ORIGIN                         | TELESCOP                       | INSTRUME
    OBJECT                         | RA                             | DEC
    EQUINOX                        | RADECSYS                       | MJD-OBS
    DATE-OBS                       | UTC                            | LST
    PI-COI                         | OBSERVER                       | ARCFILE
    DATAMD5                        | PIPEFILE                       | ESO OBS AIRM
    ESO OBS AMBI FWHM              | ESO OBS AMBI TRANS             | ESO OBS ATM
    ESO OBS CONTAINER ID           | ESO OBS CONTAINER TYPE         | ESO OBS CONTRAST
    ESO OBS DID                    | ESO OBS EXECTIME               | ESO OBS GRP
    ESO OBS ID                     | ESO OBS MOON DIST              | ESO OBS MOON FLI
    ESO OBS NAME                   | ESO OBS NTPL                   | ESO OBS OBSERVER
    ESO OBS PI-COI ID              | ESO OBS PI-COI NAME            | ESO OBS PROG ID
    ESO OBS START                  | ESO OBS STREHLRATIO            | ESO OBS TARG NAME
    ESO OBS TPLNO                  | ESO OBS TWILIGHT               | ESO OBS WATERVAPOUR
    ESO TPL DID                    | ESO TPL EXPNO                  | ESO TPL ID
    ESO TPL NAME                   | ESO TPL NEXP                   | ESO TPL PRESEQ
    ESO TPL START                  | ESO TPL VERSION                | ESO TEL AIRM END
    ESO TEL AIRM START             | ESO TEL ALT                    | ESO TEL AMBI FWHM END
    ESO TEL AMBI FWHM START        | ESO TEL AMBI IRSKY TEMP        | ESO TEL AMBI IWV END
    ESO TEL AMBI IWV START         | ESO TEL AMBI IWV30D END        | ESO TEL AMBI IWV30D START
    ESO TEL AMBI IWV30DSTD END     | ESO TEL AMBI IWV30DSTD START   | ESO TEL AMBI IWVSTD END
    ESO TEL AMBI IWVSTD START      | ESO TEL AMBI PRES END          | ESO TEL AMBI PRES START
    ESO TEL AMBI RHUM              | ESO TEL AMBI TAU0              | ESO TEL AMBI TEMP
    ESO TEL AMBI WINDDIR           | ESO TEL AMBI WINDSP            | ESO TEL AZ
    ESO TEL CHOP ST                | ESO TEL DATE                   | ESO TEL DID
    
  • ‘- -save’: This function save the list of files (without all the parameters) into a file called ‘dfitspy_file_list.txt’. An example is given below:

    ##file produced by dfitspy 2018-10-03 21:16:42.133299
    ##Current directory: /home/alien/Desktop/Test_data
    r.XSHOO.2099-14-59T09:59:57.509_tpl-A01_0000.fits
    r.XSHOO.2099-14-59T09:59:57.509_tpl-A01_0001.fits
    r.XSHOO.2099-14-59T09:59:57.509_tpl-A01_0002.fits
    r.XSHOO.2099-14-59T09:59:57.509_tpl-A01_0003.fits
    r.XSHOO.2099-14-59T09:59:57.509_tpl-A01_0004.fits
    r.XSHOO.2099-14-59T09:59:57.509_tpl-A01_0005.fits
    
  • ‘- -docs’: Display in the web browser the documentation of the code. If you have a valid internet connection it will open the online documentation, if not it will open the local documentation.

  • ‘- -version’: Display in terminal the current version of the software.