Data Aggregation Service
Astronomers now have access to an increasingly large number of online data services and archives. Querying each service can be time consuming and difficult to manage, even for small numbers of targets of interest. Each service may also differ slightly in their implementation with their own corresponding quirks and learning curve.
The Data Aggregation Service (DAS) was designed to help fill this gap. The application was built using modern asynchronous Python technologies to simultaneously retrieve query results from several services, removing the traditional bottleneck of waiting 'one-by-one' for each service.
The DAS consists of three main components:
- A set of Catalogues retrieved from a variety of services (e.g. VizieR, TAP, API, SSA)
- A set of Image collections retrieved from a variety of services (e.g. SIA, API, Observatory Archives)
- Aladin Lite to visualise images and overlay the catalogues.
Results from catalogue and image queries are cached for 24 hours to improve performance.
Please note: The current implementation is a first release intended to demonstrate the powerful capabilities offered by DAS. While it may not currently suit all users, we plan to further develop DAS into an application that can be customised to suit individual needs.
Accessing the service
The service is a web application accessible from:
You will require a Data Central account to access the service.
Currently this is used to determine if the user has access rights to the latest SkyMapper data release.
We anticipate adding user preferences and connectivity with other Data Central services (e.g. ownCloud).
If you find DAS useful, please acknowledge Data Central: Acknowledging Data Central.
There are four main parameters to DAS:
- Right Ascension (RA) in decimal degrees
- Declination (DEC) in decimal degrees
- Field-of-view (FOV) diameter in arcmin
- Error radius (ERR) in arcsec (displayed as a white circle in the Aladin Lite display)
The following restrictions are currently in place:
- FOV: Minimum of 10 arcsec. Maximum of 3 arcmin.
- ERR: Minimum of 1 arcsec. Maximum of 0.5*FOV.
These restrictions are mostly to ensure very large catalogues do not interfere too much with the user experience. We hope to alleviate these restrictions in the future once user preferences are available. This will allow, for example, magnitude limits and column name selection to reduce catalogue size.
Specifying Parameters via HTTP GET
The main parameters may be specified as HTTP GET parameters to DAS, e.g.
which returns a 1 arcmin field for the planetary nebula NGC 7009.
This approach is useful in e.g. viewing large numbers of targets in a systematic way (e.g. from a webpage or database).
Specifying Parameters via DAS
Alternatively, parameters may be entered via a web form located below the Aladin Lite display. Submitting the form will reload DAS with the new parameters.
The RA and DEC may be supplied as either:
- decimal degrees (e.g. RA=307.888365097, DEC=-7.08834222636), or
- sexigesimal format (e.g. RA=20 31 33.208, DEC=-07 05 18.03; or RA=20:31:33.208, DEC=-07:05:18.03).
The Resolve Name field allows users to conveniently query the CDS Sesame service to retrieve target coordinates.
Several catalogues are queried by DAS and loaded as a tab in the Catalogues section of DAS when ready.
The query radius for all catalogues is slightly larger than the FOV, with the exception of PhotZ/SpecZ queries which use ERR (see below).
Most catalogues can be downloaded as Excel, CSV or VOTable formats. Alternatively, tables may be sent to the TOPCAT program via SAMP. To send tables to TOPCAT, first load TOPCAT, then find and click the SAMP Enable button above the Aladin Lite display to connect to TOPCAT. Then click the SAMP links in the Catalogues summary table. TOPCAT will load the table.
The default selection of catalogues may not suit everyone, but we hope to address this by introducing user profiles in the near future. The default catalogues are:
- DCSSA - Data Central's Simple Spectral Access (SSA) service. The location of DCSSA spectra are highlighted with a rainbow wheel in the Aladin Lite display.
- DCSCS - Data Central's Simple Cone Search (SCS) service.
- PhotZ/SpecZ - Custom DAS services that query all VizieR catalogues with redshift information at the current position using the Unified Content Descriptor (UCD) for src.redshift (SpecZ) and src.redshift.phot (PhotZ). The results show redshifts (Z) collated from the specified catalogue and the corresponding column names (ZCOLS). Note that the radius for PhotZ/SpecZ queries is the DAS error radius (ERR), not the FOV like other catalogues.
- VizierTables - A list of additional VizieR catalogues that may be added by the user via the 'Load' buttons. Catalogues are matches for the target position and radius, but may give empty results in some cases.
- GAIA (I/350/gaiaedr3), HST (B/hst/obscore), ESO (B/eso/eso_arc), SDSS (V/147/sdss12), WISE (II/328/allwise), SUMSS (VIII/81B/sumss212), NVSS (VIII/65/nvss) and TIC (IV/38) are VizieR catalogues. A limit of 5000 rows each is applied.
- PS1 - PanSTARRS DR2 stack catalogue API query.
- RACS - Rapid ASKAP Continuum Survey detections from the casda.continuum_component table.
- SkyMapper - Catalogue detections from the SkyMapper TAP service. The table queried is either dr3.master or dr2.master, depending on whether Data Central user profiles are part of the SkyMapper group (required for DR3 access). The SAMP link for DR3 data is disabled for security purposes, but TOPCAT can load in the downloadable VOTable.
- SIMBAD - Query results from SIMBAD. The MAIN_ID column of the table can be clicked to open the SIMBAD page for each object.
- DECAPS, LS, DES, SMASH, DELVE - Catalogue detections from Dark Energy Camera surveys. Results are from TAP service queries on decaps_dr1.object, ls_dr8.tractor, des_dr2.main, smash_dr2.deep and delve_dr1.objects tables.
- DUST - IRSA Galactic Dust Reddening and Extinction data obtained from the program interface.
- ZTF - The Zwicky Transient Facility Public Data Release 6 obtained from the IRSA TAP table ztf_objects_dr6. Individual results may be explored via the SNAD ZTF object viewer links of the oid field.
Several image collections are queried by DAS and loaded in tabs when ready.
DAS converts all FITS images to the HiPS format before they can be displayed in Aladin Lite.
Moving your mouse over the rows in each image collection tab automatically displays the image in Aladin Lite.
Since we do not perform a full science ready reduction for most data, please be aware of the following caveats:
- Astrometry: The astrometry may be very inaccurate for some archival data, e.g. Gemini, FORS2 and HLA (to a lesser extent), as we make no corrections to the default WCS. If you require accurate astrometry, you will have to reduce your data separately.
- Photometry: The SkyMapper and FORS2 mosaics that we create via MontagePy may not be suitable for photometry since we interally use mProjectQL to ensure speedy mosaic production. This may not conserve flux in all circumstances. In other instances, check with the survey website or publication for more details.
The default image sources are:
- DCSIA - Image cutouts from Data Central's Simple Image Access (SIA) service.
- SkyMapper - 10 arcmin field-of-view mosaics are created using MontagePy in a similar fashion to this example script. Access to SkyMapper DR2 or DR3 data depends on whether Data Central user profiles are part of the SkyMapper group (required for DR3 access). Mosaics may be downloaded via the access_url links and the images links provide access to the SkyMapper input files. Please note the mosaics may not be suitable for photometry since we interally use mProjectQL to ensure speedy mosaic production.
- FORS2 - The ESO archive TAP service is queried for all VLT FORS2 images within 4.5 arcmin radius of the target position (approx. half the FORS2 FOV). The data are grouped into exposures and mosaics are created using MontagePy.
- Gemini - GMOS images from the Gemini Observatory Archive. Images taken with the multiple amplifier readout mode are merged into a single extension FITS image (no background correction is attempted).
- HLA - Hubble Legacy Archive images retrieved from the HLA SIA service. Please note that some Hubble data are very large (several hundred MB) and/or the result list may be several hundred images. For these reasons HLA images must be requested manually via the 'Load' buttons provided. We are investigating ways to improve how DAS handles large query requests.
- PS1 - Image cutouts from the PanSTARRS-1 image access service.
- DECAPS - Image cutouts from the DECam Plane Survey. Accessed using the viewer urls from the Legacy Survey website.
- DES, LS, SMASH - Image cutouts from the Simple Image Access services of the Dark Energy Camera Surveys. Data releases are DES DR2, LS DR8 and SMASH DR2.
The Aladin Lite display brings all the images and catalogues together.
Clicking the Layers button in the top left of the display shows the different image layers (surveys) available (including all of those from all the Images collections loaded by DAS) and a list of the Overlay layers (e.g. catalogues).
Catalogues may be toggled on and off in this list or using the checkboxes in the Catalogues summary tab.
The position of individual rows from catalogues are highlighted as yellow circles using the HL overlay layer.
The Overlay dropdown menu above the Aladin Lite display allows for radio surveys (RACS/NVSS/SUMSS) to appear on top of the main image. The Transparency slider adjusts the transparency of the overlay layer.
We welcome any feedback on the DAS via Data Central's Service Desk.
One of the main features we have planned is user preferences, allowing users to store their preferred data sources (catalogues and images) and other information (e.g. catalogue columns).
This current beta version is a technology demonstrator that we hope users may find useful until we can add more capabilities for customisation.
Here are some interesting objects to look at:
DAS queries a large number of different services and errors may occur when accessing them.
If a service is not accessible (e.g. server or maintenance issues), then the service will not be available to DAS. In these cases an error message will appear in red on the Catalogues or Images summary tables.
Even if a service is accessible, the individual results from that service may not be accessible. In these cases an error message will appear in red for the individual result. This is more relevant for DAS image results.
A common example here is if individual images require authentication to be accessed (e.g. FORS2 images from the ESO archive TAP service). In this case, a 401 error is typically returned. This error then appears in the status (first) column of the table of FORS2 results.
Similarly, if a data source becomes unavailable (e.g. a mounted file system becomes inaccessible), individual images may not be retrievable. In this case a 500 error may be returned.
Other errors may include parsing (ill-formed) VOTables, which are used extensively by DAS.
Under the hood
DAS is written in Python and makes use of the following technologies: