obspy.clients.fdsn.mass_downloader.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Mass Downloader for FDSN Compliant Web Services\n",
    "This package contains functionality to query and integrate data from any number of FDSN web service providers simultaneously. The package aims to formulate download requests in a way that is convenient for seismologists without having to worry about political and technical data center issues. It can be used by itself or as a library component integrated into a bigger project."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why Would You Want to Use This?\n",
    "Directly using the FDSN web services for example via the obspy.clients.fdsn client is fine for small amounts of data but quickly becomes cumbersome for larger data sets. Many data centers do provide tools to easily download larger amounts of data but that is usually only from one data center. Now most seismologists don’t really care a lot where the data they download originates - they just want the data for their use case and oftentimes they want as much data as they can get. As the number of FDSN compliant web services increases this becomes more and more cumbersome. That is where this module comes in. You\n",
    "\n",
    "1. specify the geographical region from which to download data,\n",
    "2. define a number of other restrictions (temporal, data quality, ...),\n",
    "3. and launch the download."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The mass downloader module will acquire all waveforms and associated station information across all known FDSN web service implementations producing a clean data set ready for further use. It works by\n",
    "\n",
    "1. figuring out what stations each provider offers,\n",
    "2. downloading MiniSEED and associated StationXML meta information in an efficient and data center friendly manner, and\n",
    "3. dealing with all the nasty real-world data issues like missing or incomplete data, duplicate data across data centers, e.g.\n",
    "    * Basic optional automatic quality control by assuring that the data has no-gaps/overlaps or is available for a certain percentage of the requested time span.\n",
    "    * It can relaunch download to acquire missing pieces which might happen for example if a data center has been offline.\n",
    "    * It can assure that there always is a corresponding StationXML file for the waveforms."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Usage Examples\n",
    "Before delving into the nitty-gritty details of how it works and why it does things in a certain way we’ll demonstrate the usage of this module on two annotated examples. They can serve as templates for your own needs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Earthquake Data\n",
    "The classic seismological data set consists of waveform recordings for a certain earthquake. This example downloads all data it can find for the Tohoku-Oki Earthquake from 5 minutes before the earthquake centroid time to 1 hour after. It will furthermore only download data with an epicentral distance between 70.0 and 90.0 degrees and some additional restrictions.<br> \n",
    "<span style=color:red; font-size:200%> ***Be aware that this example will attempt to download data from all FDSN data centers that ObsPy knows of and combine it into one data set.*** </span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import obspy\n",
    "from obspy.clients.fdsn.mass_downloader import RectangularDomain, \\\n",
    "    Restrictions, MassDownloader\n",
    "\n",
    "# Rectangular domain containing parts of southern Germany.\n",
    "domain = RectangularDomain(minlatitude=30, maxlatitude=50,\n",
    "                           minlongitude=5, maxlongitude=35)\n",
    "\n",
    "restrictions = Restrictions(\n",
    "    # Get data for a whole year.\n",
    "    starttime=obspy.UTCDateTime(2012, 1, 1),\n",
    "    endtime=obspy.UTCDateTime(2013, 1, 1),\n",
    "    # Chunk it to have one file per day.\n",
    "    chunklength_in_sec=86400,\n",
    "    # Considering the enormous amount of data associated with continuous\n",
    "    # requests, you might want to limit the data based on SEED identifiers.\n",
    "    # If the location code is specified, the location priority list is not\n",
    "    # used; the same is true for the channel argument and priority list.\n",
    "    network=\"BW\", station=\"A*\", location=\"\", channel=\"EH*\",\n",
    "    # The typical use case for such a data set are noise correlations where\n",
    "    # gaps are dealt with at a later stage.\n",
    "    reject_channels_with_gaps=False,\n",
    "    # Same is true with the minimum length. All data might be useful.\n",
    "    minimum_length=0.0,\n",
    "    # Guard against the same station having different names.\n",
    "    minimum_interstation_distance_in_m=100.0)\n",
    "\n",
    "# Restrict the number of providers if you know which serve the desired\n",
    "# data. If in doubt just don't specify - then all providers will be\n",
    "# queried.\n",
    "'''\n",
    "mdl = MassDownloader(providers=[\"LMU\", \"GFZ\"])\n",
    "mdl.download(domain, restrictions, mseed_storage=\"waveforms\",\n",
    "             stationxml_storage=\"stations\")\n",
    "'''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Continuous Request (Noise Data)¶\n",
    "Another use case requiring massive amounts of data are noise studies. Ambient seismic noise correlations require continuous recordings from stations over a large time span. This example downloads data, from within a certain geographical domain, for a whole year. Individual MiniSEED files will be split per day. The download helpers will attempt to optimize the queries to the data centers and split up the files again if required."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import obspy\n",
    "from obspy.clients.fdsn.mass_downloader import RectangularDomain, \\\n",
    "    Restrictions, MassDownloader\n",
    "\n",
    "# Rectangular domain containing parts of southern Germany.\n",
    "domain = RectangularDomain(minlatitude=30, maxlatitude=50,\n",
    "                           minlongitude=5, maxlongitude=35)\n",
    "\n",
    "restrictions = Restrictions(\n",
    "    # Get data for a whole year.\n",
    "    starttime=obspy.UTCDateTime(2012, 1, 1),\n",
    "    endtime=obspy.UTCDateTime(2013, 1, 1),\n",
    "    # Chunk it to have one file per day.\n",
    "    chunklength_in_sec=86400,\n",
    "    # Considering the enormous amount of data associated with continuous\n",
    "    # requests, you might want to limit the data based on SEED identifiers.\n",
    "    # If the location code is specified, the location priority list is not\n",
    "    # used; the same is true for the channel argument and priority list.\n",
    "    network=\"BW\", station=\"A*\", location=\"\", channel=\"EH*\",\n",
    "    # The typical use case for such a data set are noise correlations where\n",
    "    # gaps are dealt with at a later stage.\n",
    "    reject_channels_with_gaps=False,\n",
    "    # Same is true with the minimum length. All data might be useful.\n",
    "    minimum_length=0.0,\n",
    "    # Guard against the same station having different names.\n",
    "    minimum_interstation_distance_in_m=100.0)\n",
    "\n",
    "# Restrict the number of providers if you know which serve the desired\n",
    "# data. If in doubt just don't specify - then all providers will be\n",
    "# queried.\n",
    "'''\n",
    "mdl = MassDownloader(providers=[\"LMU\", \"GFZ\"])\n",
    "mdl.download(domain, restrictions, mseed_storage=\"waveforms\",\n",
    "             stationxml_storage=\"stations\")\n",
    "'''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Usage\n",
    "Using the download helpers requires the definition of three separate things, all of which are detailed in the following paragraphs.\n",
    "\n",
    "1. Data Selection: The data to be downloaded can be defined by enforcing geographical or temporal constraints and a couple of other options.\n",
    "2. Storage Options: Choosing where the final MiniSEED and StationXML files should be stored.\n",
    "3. Start the Download: Choose from which provider(s) to download and then launch the downloading process."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Data Selection\n",
    "Data set selection serves the purpose to limit the data to be downloaded to data useful for the purpose at hand. It is handled by two objects: subclasses of the Domain object and the Restrictions class.\n",
    "\n",
    "The domain module currently defines three different domain types used to limit the geographical extent of the queried data: RectangularDomain, CircularDomain, and GlobalDomain. Subclassing Domain enables the construction of arbitrarily complex domains. Please see the domain module for more details. Instances of these classes will later be passed to the function sparking the downloading process. A rectangular domain for example is defined like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from obspy.clients.fdsn.mass_downloader.domain import RectangularDomain\n",
    "domain = RectangularDomain(minlatitude=-10, maxlatitude=10,\n",
    "                           minlongitude=-10, maxlongitude=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Additional restrictions like temporal bounds, SEED identifier wildcards, and other things are set with the help of the Restrictions class. Please refer to its documentation for a more detailed explanation of the parameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Earthquake"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import obspy\n",
    "restrictions = Restrictions(\n",
    "    # Get data from 5 minutes before the event to one hour after the\n",
    "    # event.\n",
    "    starttime=obspy.UTCDateTime(2012, 1, 1),\n",
    "    endtime=obspy.UTCDateTime(2012, 1, 2),\n",
    "    # You might not want to deal with gaps in the data.\n",
    "    reject_channels_with_gaps=True,\n",
    "    # And you might only want waveforms that have data for at least\n",
    "    # 95 % of the requested time span.\n",
    "    minimum_length=0.95,\n",
    "    # No two stations should be closer than 10 km to each other.\n",
    "    minimum_interstation_distance_in_m=10E3,\n",
    "    # Only HH or BH channels. If a station has HH channels,\n",
    "    # those will be downloaded, otherwise the BH. Nothing will be\n",
    "    # downloaded if it has neither.\n",
    "    channel_priorities=[\"HH[ZNE]\", \"BH[ZNE]\"],\n",
    "    # Location codes are arbitrary and there is no rule as to which\n",
    "    # location is best.\n",
    "    location_priorities=[\"\", \"00\", \"10\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Noise Data\n",
    "And the restrictions for downloading a noise data set might look similar to the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import obspy\n",
    "restrictions = Restrictions(\n",
    "    # Get data for a whole year.\n",
    "    starttime=obspy.UTCDateTime(2012, 1, 1),\n",
    "    endtime=obspy.UTCDateTime(2013, 1, 1),\n",
    "    # Chunk it to have one file per day.\n",
    "    chunklength_in_sec=86400,\n",
    "    # Considering the enormous amount of data associated with\n",
    "    # continuous requests, you might want to limit the data based on\n",
    "    # SEED identifiers. If the location code is specified, the\n",
    "    # location priority list is not used; the same is true for the\n",
    "    # channel argument and priority list.\n",
    "    network=\"BW\", station=\"A*\", location=\"\", channel=\"BH*\",\n",
    "    # The typical use case for such a data set are noise correlations\n",
    "    # where gaps are dealt with at a later stage.\n",
    "    reject_channels_with_gaps=False,\n",
    "    # Same is true with the minimum length. Any data during a day\n",
    "    # might be useful.\n",
    "    minimum_length=0.0,\n",
    "    # Sanitize makes sure that each MiniSEED file also has an\n",
    "    # associated StationXML file, otherwise the MiniSEED files will\n",
    "    # be deleted afterwards. This is not desirable for large noise\n",
    "    # data sets.\n",
    "    sanitize=False,\n",
    "    # Guard against the same station having different names.\n",
    "    minimum_interstation_distance_in_m=100.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The network, station, location, and channel codes are directly passed to the station service of each fdsn-ws implementation and can thus take comma separated string lists as arguments, i.e."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "restrictions = Restrictions(\n",
    "    ...\n",
    "    network=\"BW,G?\", station=\"A*,B*\",\n",
    "    ...\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not all fdsn-ws implementations support the direct exclusion of network or station codes. The exclude_networks and exclude_stations arguments should thus be used for that purpose to ensure compatibility across all data providers, e.g."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "restrictions = Restrictions(\n",
    "    ...\n",
    "    network=\"B*,G*\", station=\"A*, B*\",\n",
    "    exclude_networks=[\"BW\", \"GR\"],\n",
    "    exclude_stations=[\"AL??\", \"*O\"],\n",
    "    ...\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is also possible to restrict the downloaded stations to stations part of an existing inventory object which can originate from a StationXML file or from other sources. It will only keep stations that are part of the inventory object. Channels are still selected dynamically based on the other restrictions.<br><br> Keep in mind that all other restrictions still apply - passing an inventory will just further restrict the possibly downloaded data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "restrictions = Restrictions(\n",
    "    ...\n",
    "    limit_stations_to_inventory=inv,\n",
    "    ...\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Storage Options\n",
    "After determining what to download, the helpers must know where to store the requested data. That requires some flexibility in case the mass downloader is to be integrated as a component into a bigger system. An example is a toolbox that has a database to manage its data.\n",
    "\n",
    "A major concern is to not download pre-existing data. In order to enable such a use case the download helpers can be given functions that are evaluated when determining the file names of the requested data. Depending on the return value, the helper class will download the whole, part, or even none, of that particular piece of data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Storing MiniSEED waveforms\n",
    "The MiniSEED storage rules are set by the mseed_storage argument of the download() method of the MassDownloader class"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Option 1: Folder Name**\n",
    "In the simplest case it is just a folder name:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mseed_storage = \"waveforms\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will cause all MiniSEED files to be stored as\n",
    "\n",
    "*waveforms/NETWORK.STATION.LOCATION.CHANNEL__STARTTIME__ENDTIME.mseed.*\n",
    "\n",
    "An example of this is\n",
    "\n",
    "*waveforms/BW.FURT..BHZ__20141027T163723Z__20141027T163733Z.mseed*\n",
    "\n",
    "which is rather general but also quite long."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Option 2: String Template**\n",
    "\n",
    "For more control use the second possibility and provide a string containing {network}, {station}, {location}, {channel}, {starttime}, and {endtime} format specifiers. These values will be interpolated to acquire the final filename. The start and end times will be formatted with strftime() with the specifier \"%Y%m%dT%H%M%SZ\" in an effort to avoid colons which are troublesome in file names on many systems."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mseed_storage = (\"some_folder/{network}/{station}/\"\n",
    "                 \"{channel}.{location}.{starttime}.{endtime}.mseed\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "results in\n",
    "\n",
    "*some_folder/BW/FURT/BHZ..20141027T163723Z.20141027T163733Z.mseed.*\n",
    "\n",
    "The download helpers will create any non-existing folders along the path."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Option 3: Custom Function**\n",
    "\n",
    "The most complex but also most powerful possibility is to use a function which will be evaluated to determine the filename. **If the function returns** True , **the MiniSEED file is assumed to already be available and will not be downloaded again; keep in mind that in that case no station data will be downloaded for that channel.** If it returns a string, the MiniSEED file will be saved to that path. Utilize closures to use any other parameters in the function. This hypothetical function checks if the file is already in a database and otherwise returns a string which will be interpreted as a filename."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_mseed_storage(network, station, location, channel, starttime,\n",
    "                      endtime):\n",
    "    # Returning True means that neither the data nor the StationXML file\n",
    "    # will be downloaded.\n",
    "    if is_in_db(network, station, location, channel, starttime, endtime):\n",
    "        return True\n",
    "    # If a string is returned the file will be saved in that location.\n",
    "    return os.path.join(ROOT, \"%s.%s.%s.%s.mseed\" % (network, station,\n",
    "                                                     location, channel))\n",
    "mseed_storage = get_mseed_storage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=color:blue; font-size:200%>Note<br>\n",
    "No matter which approach is chosen, if a file already exists, it will not be overwritten; it will be parsed and the download helper class will attempt to download matching StationXML files.</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Storing StationXML files\n",
    "The same logic applies to the StationXML files. This time the rules are set by the stationxml_storage argument of the download() method of the MassDownloader class. StationXML files will be downloaded on a per-station basis thus all channels and locations from one station will end up in the same StationXML file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Option 1: Folder Name**\n",
    "\n",
    "A simple string will be interpreted as a folder name. This example will save the files to \"stations/NETWORK.STATION.xml\", e.g. to \"stations/BW.FURT.xml\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stationxml_storage = \"stations\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Option 2: String Template**\n",
    "\n",
    "Another option is to provide a string formatting template, e.g."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stationxml_storage = \"some_folder/{network}/{station}.xml\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "will write to *\"some_folder/NETWORK/STATION.xml\"*\n",
    "\n",
    ", in this case for example to \"some_folder/BW/FURT.xml\"*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=color:blue; font-size:200%>Note<br>\n",
    "If the StationXML file already exists, it will be opened to see what is in the file. In case it does not contain all necessary channels, it will be deleted and **only those channels needed in the current run will be downloaded again.** Pass a custom function to the stationxml_path argument if you require different behavior as documented in the following section.</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Option 3: Custom Function**\n",
    "\n",
    "As with the waveform data, the StationXML paths can also be set with the help of a function. The function in this case is a bit more complex than for the waveform case. It has to return a dictionary with three keys: \"available_channels\", \"missing_channels\", and \"filename\". \"available_channels\" is a list of channels that are already available as station information and that require no new download. Make sure to include all already available channels; this information is later used to discard MiniSEED files that have no corresponding station information. \"missing_channels\" is a list of channels for that particular station that must be downloaded and \"filename\" determines where to save these. Please note that in this particular case the StationXML file will be overwritten if it already exists and only the \"missing_channels\" will be downloaded to it, independent of what already exists in the file.\n",
    "\n",
    "Alternatively the function can also return a string and the behaviour is the same as two first options for the stationxml_storage argument.\n",
    "\n",
    "The next example illustrates a complex use case where the availability of each channel’s station information is queried in some database and only those channels that do not exist yet will be downloaded. Use closures to pass more arguments to the function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_stationxml_storage(network, station, channels, starttime, endtime):\n",
    "    available_channels = []\n",
    "    missing_channels = []\n",
    "    for location, channel in channels:\n",
    "        if is_in_db(network, station, location, channel, starttime,\n",
    "                    endtime):\n",
    "            available_channels.append((location, channel))\n",
    "        else:\n",
    "            missing_channels.append((location, channel))\n",
    "    filename = os.path.join(ROOT, \"%s.%s.xml\" % (network, station))\n",
    "    return {\n",
    "        \"available_channels\": available_channels,\n",
    "        \"missing_channels\": missing_channels,\n",
    "        \"filename\": filename}\n",
    "stationxml_storage = get_stationxml_storage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3: Start the Download¶\n",
    "The final step is to actually start the download. Pass the previously created domain, restrictions, and path settings and off you go. Two more parameters of interest are the chunk_size_in_mb setting which controls how much data is requested per thread, client and request. threads_per_clients control how many threads are used to download data in parallel per data center - 3 is a value in agreement with some data centers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mdl = MassDownloader()  \n",
    "mdl.download(domain, restrictions, chunk_size_in_mb=50,\n",
    "             threads_per_client=3, mseed_storage=mseed_storage,\n",
    "             stationxml_storage=stationxml_storage)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}