2.2.1   Deployment¶

Qserv was successfully deployed on GKE as a StatefulSet with persistent volumes. GKE worked well as a testbed to improve the Kubernetes configuration for all deployment environments. Qserv shards its data into chunks which are then distributed across multiple nodes. Some queries access only a single chunk; others can query across all chunks in parallel. The same number of nodes (30) and chunking scheme was initially used on GKE as had been used for tests on the LSST Prototype Data Access Center (PDAC) hardware at NCSA as reported in DMTR-52. Later, the Qserv replication service, which had not been previously been deployed under Kubernetes, was also successfully integrated, and that service was used to elastically redistribute chunks when the cluster was expanded from 30 nodes to 40.

2.2.2   Performance¶

Performance was tested using Google Cloud’s Persistent Disk block storage. This is a relatively expensive permanent store on network-attached disk.

Queries were executed on three separate systems: the PDAC at NCSA (from DMTR-52), Qserv on Google Cloud, and the Google BigQuery database service. Results are given in Document-31100.

Query times for “point” queries limited to a single chunk and sometimes indexed were generally faster in the cloud than on the PDAC hardware. But “scan” queries that processed large amounts of data were slower. There is one data point that was not included in the above document: a COUNT(*) query on the wise_00.allwise_p3as_psd table took 1141.1 seconds in a “typical” run on the cloud, as opposed to the reported maximum of 1825.9 seconds. This suggests that anecdotal but unrecorded measurements of approximately 20% degradation in query times for larger queries are reasonable.

Comparisons with other types of cloud storage were not done. Temporary node-attached storage, while perhaps higher performance, would be unsuitable for a production deployment as there would be insufficient space and permanence. Google Cloud Storage object store would be cheaper but would require major code modifications within Qserv.

The results for BigQuery show significant speedups for queries that retrieve a limited number of columns, as expected due to BigQuery’s columnar organization. Spherical geometry primitives were able to be adapted for use in astronomical queries. Proper data organization, in particular clustering the BigQuery tables by spatial index, along with the use of a spatial restriction primitive led to substantial improvements in query time for a near-neighbor query. Retrieval of individual objects was relatively slow, however, due to BigQuery’s startup time and lack of indexing. It seems clear that it is possible, with some work on ADQL (Astronomical Data Query Language) translation and possibly creation of auxiliary tables, for BigQuery to handle the largest-scale catalog queries.

Possible alternatives to BigQuery include BigTable, a non-relational database that might have more difficulties with ADQL translation and joins, and Spanner, a SQL relational database that might be too featureful and expensive for LSST’s needs. Neither was investigated during the engagement.

Cloud Dataproc, a hosted Apache Spark and Hadoop data processing service, was not tested, but there is no reason to believe that it will not work as advertised against data stored in BigQuery.

3.2.1   Data Transfer¶

Images were transmitted from a data transfer node in the AURA machine room in La Serena to a US storage bucket within Google Cloud Storage. The configuration of the node, networks, and the data are given in IT-991 and DM-18125; the latter contains the measurements obtained. The fastest network link available from La Serena to Santiago (where peering with Google’s own network occurred) was a 10 Gbit/sec link. As a result, the data to be transferred was scaled down appropriately. Nevertheless, the results are not fully representative of the performance of the 100 Gbit/sec link that will be available for LSST Operations as there may be downstream bottlenecks, effects from multiple parallel transfer nodes, problems from large bandwidth-delay products, etc.

The Google gsutil tool was used to perform the copy.

Simple regression over the 4 measured data points gives a large startup time of 5.6 sec, even with data in memory. The transfer bandwidth derived from the regression (1500 Mbits/sec) is quite reasonable given the lack of tuning. Overall, the results indicate that the Santiago-to-Google Cloud networking can handle large transfers, although it is yet to be proven that ten times the scale could be handled on a production basis. Substantial further work would likely be required to reduce the transfer latency, where the goal is under 2 sec, if this were to be used as the primary channel for Prompt Processing image transfer.

3.2.2   PPDB¶

The PPDB, in a PostgreSQL implementation, was successfully deployed on the cloud on a single large compute instance. Its performance was tested using existing client code. The results are documented in DMTN-113. On the cloud, it was possible to execute a more realistic scenario than on previous development hardware. The client code could run on a separate machine from the database, and the database itself could run on a larger server (64 vCPUs versus 56 hyperthreads shared with clients; 10 TB SSD versus 2 TB NVMe + 3 TB SATA SSD and 7.3 TB RAID).

The performance was found to be roughly comparable with Oracle RAC, somewhat worse for writes/inserts. With the larger machine size, it was possible to extend the PostgreSQL results to ~2 months of visits versus ~2 weeks on the previous hardware.

3.2.3   Alert Distribution and Production¶

These goals were not attempted. Existing tests were deemed to be sufficient to show the Alert Distribution scaling. The pipeline code was not in a readily-deployable state for Alert Produciton testing.