Data Central uses cookies to make your browsing experience better. By using Data Central you agree to its use of cookies. Learn more I agree

Documentation

Find information, examples, FAQs and extensive descriptions of the data, curated by the survey teams.

Docs ref

Reference

A complete guide to using Data Central
Feb. 21, 2018 by S. O'Toole
•
March 7, 2023, 2:48 p.m. B. Miszalski

TAP Service

During August and September 2022 we updated our TAP service for Data Central.

The service may be accessed from the endpoint at https://datacentral.org.au/vo/tap.

Several tables are available and more will be added over time.

The new service is based upon VOLLT/TAP-Lib which is achieved using the Dockerised implementation of https://github.com/gmantele/vollt-tap-docker.

Getting Started

The TAP service uses ADQL as the query language. It is similar to SQL and more info can be found here.

It's easy to get started with the TAP service via TOPCAT:

  • Go to the VO menu of TOPCAT and choose Table Access Protocol (TAP) Query.
  • Enter into Selected TAP Service at the bottom of the window https://datacentral.org.au/vo/tap
  • Click the Use Service button.
  • You should now see a list of available tables to query.

Current Tables

We are still in the process of adding tables from various Data Central hosted surveys.

The main tables added so far belong to:

  • 2dfgrs_fdr
  • 2mass_fdr
    • Point Source Catalogue (PSC),
    • Scan Information Table (SCN), and
    • Extended Source Catalogue (XSC)
  • 6dfgs_fdr
  • g4jy_dr1
  • galah_dr1, galah_dr2, galah_dr3
  • gama_dr2
  • gleamx_dr1
  • lobes_dr1
  • s5_dr1
  • sami_dr1, sami_dr2, sami_dr3
  • wigglez_final

Other tables may be present for testing purposes.

Warning: 29 rows from the 6dfgs_fdr SPECTRA table had problematic comment fields that resulted in an excess number of columns. These were trimmed as necessary to ingest into TAP but there may remain issues with certain rows. This issue affects SPECID values of 7788, 15445, 18050, 18090, 21104, 28675, 32883, 43552, 84515, 84545, 95045, 97109, 97110, 97185, 97186, 97189, 97190, 97191, 97192, 97193, 97196, 97198, 97228, 100718, 103804, 104160, 104190, 104192 and 115431.

Basic information about existing tables may be obtained by querying the TAP_SCHEMA tables.

These are special tables belonging to each TAP service that record all the table metadata.

For example, the following gives a quick summary of what datasets are available:

SELECT * FROM TAP_SCHEMA.schemas

To list all the tables in gama_dr2:

SELECT * FROM TAP_SCHEMA.tables WHERE table_name LIKE 'gama_dr2%'

The best way to interact with the TAP service tables is via the TAP service. However, if you are just interested in getting a csv copy of the tables you could run the following Python code to generate a script to download each table. Note the format changes in each url. In the first instance we convert the votable format results to a pandas dataframe, while in the second we request csv format:

from urllib.parse import quote
from io import BytesIO
import requests
from astropy.io.votable import parse_single_table, parse
query = """select * from TAP_SCHEMA.tables WHERE schema_name='gama_dr2'"""
url = "https://datacentral.org.au/vo/tap/sync?REQUEST=doQuery&lang=ADQL&FORMAT=votable&QUERY=" + quote(query)
df = parse_single_table(BytesIO(requests.get(url).content)).to_table(use_names_over_ids=True).to_pandas()
for idx, row in df.iterrows():
    print (row['table_name'])
    query = """select * from %s""" % (row['table_name'])
    url = "https://datacentral.org.au/vo/tap/sync?REQUEST=doQuery&lang=ADQL&FORMAT=csv&QUERY=" + quote(query)
    print ("wget \"%s\" -O %s.csv" % (url,row['table_name']))

Quoting

If the schema_name or table_name of your table of interest starts with a number, you will need to encapsulate that part of the table name in quotes. For example:

SELECT TOP 10 * FROM "2mass_fdr".psc

Another example:

SELECT TOP 10 * FROM noirlab_test2."1_allwise"

Table Uploads

Perhaps the most important feature of the new TAP service is the support for table uploads.

Uploaded tables can be used in ADQL queries in conjunction with existing tables for multiple purposes including cross-matching.

The tables exist only for the purposes of your query and are not ingested into the TAP service.

Currently the uploaded table size is limited to 10GB or 10 million rows.

Uploads may be achieved via TOPCAT (search the TOPCAT documentation for TAP_UPLOAD) or via Python (e.g. using pyvo or astroquery).

Multiple tables may be uploaded at the same time.

In the following we add some basic examples of using the upload functionality from Python. We intend to add more detailed examples in the future.

Uploading a table using TapPlus (astroquery)

from astroquery.utils.tap.core import TapPlus
adc = TapPlus(url="https://datacentral.org.au/vo/tap")
upload_me = "table.xml"
adql = """SELECT * FROM tap_upload.table_test""" 
j=adc.launch_job_async(query=adql,upload_resource=upload_me,upload_table_name="table_test",verbose=True,dump_to_file=True) 
r = j.get_results()
r.pprint()

Uploading a table using pyvo

import pyvo as vo
tap_service = vo.dal.TAPService("https://datacentral.org.au/vo/tap")
upload_me = "table.xml"
adql = """SELECT * FROM tap_upload.table_test"""
uploads = {"table_test" : upload_me}
tap_results = tap_service.search(adql,uploads=uploads)
tap_results.to_table().write('output.xml',format='votable')
print (tap_results)

Alternative table upload using pyvo

By default astroquery and pyvo load the result from async TAP queries into memory.

If your TAP query result is several GB or more in size, then this could be problematic on a system with modest memory.

In this example we perform upload a table and perform an async query, but only retrieve the url.

The url may be downloaded to a file on disk, or as in this example, into memory in Python for later use.

import pyvo as vo
from io import BytesIO
import requests
from astropy.io.votable import parse_single_table, parse

url = "https://datacentral.org.au/vo/tap"
adql = """SELECT * FROM tap_upload.table_test""" 
upload_me = "table.xml" 
uploads = {"table_test" : upload_me}
job = vo.dal.AsyncTAPJob.create(url, query=adql, language="ADQL", maxrec=None, uploads=uploads,session=None)
job = job.run().wait()
url = job.result_uri
#print out url 
print (url)
#up to this point we have not downloaded the table or loaded it into memory
#if you have the memory available, you could also download this votable
#into memory and open with astropy
table = parse_single_table(BytesIO(requests.get(url).content)).to_table(use_names_over_ids=True)
print (table)
#convert to pandas
df = table.to_pandas()
print (df.shape)

Common operations and User Defined Functions

We have added the q3c extension to the database underpinning the TAP service.

The q3c functions allow users to perform complex spatial queries including cone searches and table cross-matching.

In the following we give several examples of using q3c queries to perform some of these common operations.

Cone Search

The following uses q3c_radial_query to perform a cone search of 0.2 deg radius at ra=172.92119, dec=-25.590649 of the GLEAM-X DR1 catalogue. A new (first) column is added, called dist, that uses the q3c_dist to calculate the distance in arcseconds of the results from the query position.

SELECT q3c_dist(RAJ2000,DEJ2000, 172.92119,-25.590649)*3600 AS dist, * 
FROM gleamx_dr1.idr1_subset_filtered_seds_paper 
WHERE 't' = q3c_radial_query(RAJ2000,DEJ2000,172.92119,-25.590649,0.2) 
ORDER BY dist ASC
Feb. 21, 2018 by S. O'Toole
•
March 7, 2023, 2:48 p.m. B. Miszalski