Data Aggregation Service
Background
Astronomers now have access to an increasingly large number of online data services and archives. Querying each service can be time consuming and difficult to manage, even for small numbers of targets of interest. Each service may also differ slightly in their implementation with their own corresponding quirks and learning curve.
The Data Aggregation Service (DAS) was designed to help fill this gap. The application was built using modern asynchronous Python technologies to simultaneously retrieve query results from several services, removing the traditional bottleneck of waiting 'one-by-one' for each service.
The DAS consists of three main components:
- A set of Catalogues retrieved from a variety of services (e.g. VizieR, TAP, API, SSA)
- A set of Image collections retrieved from a variety of services (e.g. SIA, API, Observatory Archives)
- Aladin Lite to visualise images and overlay the catalogues.
Results from catalogue and image queries are cached for 24 hours to improve performance.
Please note: The current implementation is a first release intended to demonstrate the powerful capabilities offered by DAS. While it may not currently suit all users, we plan to further develop DAS into an application that can be customised to suit individual needs.
Accessing the service
The service is a web application accessible from:
https://das.datacentral.org.au/
You will require a Data Central account to access the service.
Currently this is used to determine if the user has access rights to the latest SkyMapper data release.
We anticipate adding user preferences and connectivity with other Data Central services (e.g. ownCloud).
Acknowledging DAS
Any work published using the Data Aggregation Service should add the acknowledgements for DAS located at this link.
Input Parameters
Main Parameters
There are four main parameters to DAS:
- Right Ascension (RA) in decimal degrees
- Declination (DEC) in decimal degrees
- Field-of-view (FOV) diameter in arcmin
- Error radius (ERR) in arcsec (displayed as a white circle in the Aladin Lite display)
The following restrictions are currently in place:
- FOV: Minimum of 10 arcsec. Maximum of 3 arcmin.
- ERR: Minimum of 1 arcsec. Maximum of 0.5*FOV.
These restrictions are mostly to ensure very large catalogues do not interfere too much with the user experience. We hope to alleviate these restrictions in the future once user preferences are available. This will allow, for example, magnitude limits and column name selection to reduce catalogue size.
Additional parameters may be specified to control which data sources are loaded.
These are covered in the following section.
Specifying Additional Parameters via HTTP GET
The main parameters may be specified as HTTP GET parameters to DAS, e.g.
https://das.datacentral.org.au/das?RA=316.0450639&DEC=-11.36349495&FOV=1.0&ERR=10.0
which returns a 1 arcmin field for the planetary nebula NGC 7009.
This approach is useful in e.g. viewing large numbers of targets in a systematic way (e.g. from a webpage or database).
As of April 2022, you can specify the following optional HTTP GET parameters:
- OVERLAY=racs or OVERLAY=nvss or OVERLAY=sumss or OVERLAY=emu_pilot - sets the overlay of the survey to be active
- OVERLAYTRANSP=0.6 (float between 0.0 and 1.0) - set the transparency of the overlay layer (only works if OVERLAY is also specified)
- CAT or IMG or VIZ. See the following table for accepted parameters:
If there are services you'd like DAS to query that are not included below and not on VizieR, please contact us.
| Parameter | Description | Parameter | Description | Parameter | Description |
|---|---|---|---|---|---|
| CAT=0 | Turn off all catalogue sources. This overrides all other CAT parameters. | IMG=0 | Turn off all image sources. This overrides all other IMG parameters. | VIZ=name,cat | Load CDS VizieR catalogue 'cat' and use 'name' as the catalogue and tab name. The name will appear capitalised in DAS. |
| CAT=DEFAULT | Load default catalogue sources. Required if you want to load the default sources PLUS additional ones (e.g. adding VizieR catalogues to the default). | For example, DAS loads the following default VizieR catalogues: | |||
| CAT=DCSCS | Load Data Central Simple Cone Search (SCS) catalogue | IMG=DCSIA | Load Data Central SIA images | VIZ=eso,B/eso/eso_arc | |
| CAT=DCSSA | Load Data Central Simple Spectral Access (SSA) catalogue | IMG=DECAPS | Load DECam Plane Survey (DECAPS) API cutout images | VIZ=hst,B/hst/obscore | |
| CAT=DECAPS | Load DECam Plane Survey (DECAPS) NOIRLab TAP catalogue | IMG=DES | Load Dark Energy Survey (DES) NOIRLab SIA images | VIZ=gaia,I/350/gaiaedr3 | |
| CAT=DELVE | Load The DECam Local Volume Exploration Survey (DELVE) NOIRLab TAP catalogue | IMG=FORS2 | Load ESO FORS2 mosaics generated on-demand by DAS | VIZ=sdss,V/147/sdss12 | |
| CAT=DES | Load Dark Energy Survey (DES) NOIRLab TAP catalogue | IMG=GMOS | Load Gemini GMOS mosaics generated on-demand by DAS | VIZ=wise,II/328/allwise | |
| CAT=DUST | Load the IRSA Galactic Dust Reddening and Extinction table | IMG=HLA | Load Hubble Legacy Archive (HLA) SIA images | VIZ=sumss,VIII/81B/sumss212 | |
| CAT=FINK | Load the FINK transient broker conesearch catalogue | IMG=LS | Load Legacy Survey (LS) NOIRLab SIA images | VIZ=nvss,VIII/65/nvss | |
| CAT=LS | Load Legacy Survey (LS) NOIRLab TAP catalogue | IMG=MUSE | Load MUSE Ancillary Whitelight Images from ESO Phase3 Data Product Archive | VIZ=2mass,II/246 | |
| CAT=PS1 | Load the Pan-STARRS API catalogue | IMG=PS1 | Load Pan-STARRS cutout images | VIZ=tic,IV/38 | |
| CAT=PZ | Load the Photometric Redshift catalogue aggregated by DAS | IMG=SM | Load SkyMapper mosaics generated on-demand by DAS | ||
| CAT=RACS | Load the RAPID ASKAP Continuum Survey (RACS) TAP catalogue | IMG=SMASH | Load The Survey of the Magellanic Stellar History (SMASH) NOIRLab SIA images | ||
| CAT=SIMBAD | Load the CDS SIMBAD catalogue | IMG=VHS | Load VISTA Hemisphere Survey (VHS) tilestack image cutouts from the VISTA Science Archive (VSA) | ||
| CAT=SM | Load the SkyMapper TAP catalogue | IMG=VIKING | Load VISTA Kilo-degree Infrared Galaxy Survey (VIKING) tilestack image cutouts from the VISTA Science Archive (VSA) | ||
| CAT=SMASH | Load The Survey of the Magellanic Stellar History (SMASH) NOIRLab TAP catalogue | ||||
| CAT=SZ | Load the Spectroscopic Redshift catalogue aggregated by DAS | ||||
| CAT=VHS | Load the VISTA Hemisphere Survey (VHS) NOIRLab TAP catalogue | ||||
| CAT=VT | Load the CDS VizieR table catalogue | ||||
| CAT=ZTF | Load the Zwicky Transient Facility (ZTF) IRSA TAP catalogue |
Examples of GET parameter usage
- Load only image sources: https://das.datacentral.org.au/das?CAT=0
- Load only image sources and the Acker catalogue of Planetary Nebulae: https://das.datacentral.org.au/das?VIZ=acker,V/84
- Load the DEFAULT catalogue sources and the Acker catalogue: https://das.datacentral.org.au/das?CAT=DEFAULT&VIZ=acker,V/84
- Load only the FINK and ZTF catalogues: https://das.datacentral.org.au/das?CAT=ZTF&CAT=FINK&IMG=0
Specifying Parameters via DAS
Alternatively, parameters may be entered via a web form located below the Aladin Lite display. Submitting the form will reload DAS with the new parameters.
The RA and DEC may be supplied as either:
- decimal degrees (e.g. RA=307.888365097, DEC=-7.08834222636), or
- sexigesimal format (e.g. RA=20 31 33.208, DEC=-07 05 18.03; or RA=20:31:33.208, DEC=-07:05:18.03).
The Resolve Name field allows users to conveniently query the CDS Sesame service to retrieve target coordinates.
Catalogues
Several catalogues are queried by DAS and loaded as a tab in the Catalogues section of DAS when ready.
The query radius for all catalogues is slightly larger than the FOV, with the exception of PhotZ/SpecZ queries which use ERR (see below).
Most catalogues can be downloaded as Excel, CSV or VOTable formats. Alternatively, tables may be sent to the TOPCAT program via SAMP. To send tables to TOPCAT, first load TOPCAT, then find and click the SAMP Enable button above the Aladin Lite display to connect to TOPCAT. Then click the SAMP links in the Catalogues summary table. TOPCAT will load the table.
The default catalogues are:
- DCSSA - Data Central's Simple Spectral Access (SSA) service.
- DCSCS - Data Central's Simple Cone Search (SCS) service.
- PhotZ/SpecZ - Custom DAS services that query all VizieR catalogues with redshift information at the current position using the Unified Content Descriptor (UCD) for src.redshift (SpecZ) and src.redshift.phot (PhotZ). The results show redshifts (Z) collated from the specified catalogue and the corresponding column names (ZCOLS). Note that the radius for PhotZ/SpecZ queries is the DAS error radius (ERR), not the FOV like other catalogues.
- VizierTables - A list of additional VizieR catalogues that may be added by the user via the 'Load' buttons. Catalogues are matches for the target position and radius, but may give empty results in some cases.
- GAIA (I/350/gaiaedr3), HST (B/hst/obscore), ESO (B/eso/eso_arc), SDSS (V/147/sdss12), 2MASS (II/246), WISE (II/328/allwise), SUMSS (VIII/81B/sumss212), NVSS (VIII/65/nvss) and TIC (IV/38) are VizieR catalogues. A limit of 5000 rows each is applied.
- PS1 - PanSTARRS DR2 stack catalogue API query.
- RACS - Rapid ASKAP Continuum Survey detections from the casda.continuum_component table.
- SkyMapper - Catalogue detections from the SkyMapper TAP service. The table queried is either dr3.master or dr2.master, depending on whether Data Central user profiles are part of the SkyMapper group (required for DR3 access). The SAMP link for DR3 data is disabled for security purposes, but TOPCAT can load in the downloadable VOTable.
- SIMBAD - Query results from SIMBAD. The MAIN_ID column of the table can be clicked to open the SIMBAD page for each object.
- DECAPS, LS, DES, SMASH, DELVE - Catalogue detections from Dark Energy Camera surveys. Results are from the NOIRLab TAP service queries on decaps_dr1.object, ls_dr9.tractor, des_dr2.main, smash_dr2.deep and delve_dr2.objects tables.
- VHS - Catalogue detections from the VISTA Hemisphere Survey (VHS) DR5. Results are from the NOIRLab TAP service queries on the vhs_dr5.vhs_cat_v3 table. The NOIRLab VHS catalogue is a copy of the ESO hosted catalogue. See here for more details.
- DUST - IRSA Galactic Dust Reddening and Extinction data obtained from the program interface.
- ZTF - The Zwicky Transient Facility Public Data Release 10 obtained from the IRSA TAP table ztf_objects_dr10. Individual results may be explored via the SNAD ZTF object viewer links of the oid field.
- FINK - A cone search of the FINK transient broker database.
Images
Several image collections are queried by DAS and loaded in tabs when ready.
In some special cases we create our own mosaics using MontagePy (SkyMapper and FORS2) or astropy (Gemini).
DAS converts all FITS images to the HiPS format before they can be displayed in Aladin Lite.
Moving your mouse over the rows in each image collection tab automatically displays the image in Aladin Lite.
Since we do not perform a full science ready reduction for most data, please be aware of the following caveats:
- Astrometry: The astrometry may be very inaccurate for some archival data, e.g. Gemini, FORS2 and HLA (to a lesser extent), as we make no corrections to the default WCS. If you require accurate astrometry, you will have to reduce your data separately.
- Photometry: The SkyMapper and FORS2 mosaics that we create via MontagePy may not be suitable for photometry since we interally use mProjectQL to ensure speedy mosaic production. This may not conserve flux in all circumstances. In other instances, check with the survey website or publication for more details.
The default image sources are:
- DCSIA - Image cutouts from Data Central's Simple Image Access (SIA) service.
- SkyMapper - 10 arcmin field-of-view mosaics are created using MontagePy in a similar fashion to this example script. Access to SkyMapper DR2 or DR3 data depends on whether Data Central user profiles are part of the SkyMapper group (required for DR3 access). Mosaics may be downloaded via the access_url links and the images links provide access to the SkyMapper input files. Please note the mosaics may not be suitable for photometry since we interally use mProjectQL to ensure speedy mosaic production.
- FORS2 - The ESO archive TAP service is queried for all VLT FORS2 images within 4.5 arcmin radius of the target position (approx. half the FORS2 FOV). The data are grouped into exposures and mosaics are created using MontagePy.
- Gemini - GMOS images from the Gemini Observatory Archive. Images taken with the multiple amplifier readout mode are merged into a single extension FITS image (no background correction is attempted).
- HLA - Hubble Legacy Archive images retrieved from the HLA SIA service. Please note that some Hubble data are very large (several hundred MB) and/or the result list may be several hundred images. For these reasons HLA images must be requested manually via the 'Load' buttons provided. We are investigating ways to improve how DAS handles large query requests.
- PS1 - Image cutouts from the PanSTARRS-1 image access service.
- DECAPS - Image cutouts from the DECam Plane Survey. Accessed using the viewer urls from the Legacy Survey website.
- DES, LS, SMASH - Image cutouts from the Simple Image Access services of the Dark Energy Camera Surveys. Data releases are DES DR2, LS DR8 and SMASH DR2.
Aladin Lite
The Aladin Lite display brings all the images and catalogues together.
Clicking the Layers button in the top left of the display shows the different image layers (surveys) available (including all of those from all the Images collections loaded by DAS) and a list of the Overlay layers (e.g. catalogues).
Catalogues may be toggled on and off in this list or using the checkboxes in the Catalogues summary tab.
The summary tab also shows the shape and colour of marker used for each catalogue.
The position of individual rows from catalogues are highlighted as yellow circles using the HL overlay layer.
The Overlay dropdown menu above the Aladin Lite display allows for radio surveys (RACS/NVSS/SUMSS) to appear on top of the main image. The Transparency slider adjusts the transparency of the overlay layer.
Future Plans
We welcome any feedback on the DAS via Data Central's Service Desk.
One of the main features we have planned is user preferences, allowing users to store their preferred data sources (catalogues and images) and other information (e.g. catalogue columns).
This current beta version is a technology demonstrator that we hope users may find useful until we can add more capabilities for customisation.
Sample queries
Here are some interesting objects to look at:
Error handling
DAS queries a large number of different services and errors may occur when accessing them.
If a service is not accessible (e.g. server or maintenance issues), then the service will not be available to DAS. In these cases an error message will appear in red on the Catalogues or Images summary tables.
Even if a service is accessible, the individual results from that service may not be accessible. In these cases an error message will appear in red for the individual result. This is more relevant for DAS image results.
A common example here is if individual images require authentication to be accessed (e.g. FORS2 images from the ESO archive TAP service). In this case, a 401 error is typically returned. This error then appears in the status (first) column of the table of FORS2 results.
Similarly, if a data source becomes unavailable (e.g. a mounted file system becomes inaccessible), individual images may not be retrievable. In this case a 500 error may be returned.
Other errors may include parsing (ill-formed) VOTables, which are used extensively by DAS.
Under the hood
DAS is written in Python and makes use of the following technologies: