{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Examples of pyesgf download usage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Obtain MyProxy credentials to allow downloading files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyesgf.logon import LogonManager\n",
    "lm = LogonManager()\n",
    "lm.logoff()\n",
    "lm.is_logged_on()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "myproxy_host = 'esgf-data.dkrz.de'\n",
    "lm.logon(username=None, password=None, hostname=myproxy_host)\n",
    "lm.is_logged_on()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now download a file using the ESGF wget script extracted from the server:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyesgf.search import SearchConnection\n",
    "conn = SearchConnection('https://esgf-data.dkrz.de/esg-search', distrib=False)\n",
    "ctx = conn.new_context(project='obs4MIPs', institute='FUB-DWD')\n",
    "ds = ctx.search()[0]\n",
    "\n",
    "import tempfile\n",
    "fc = ds.file_context()\n",
    "wget_script_content = fc.get_download_script()\n",
    "script_path = tempfile.mkstemp(suffix='.sh', prefix='download-')[1]\n",
    "with open(script_path, \"w\") as writer:\n",
    "    writer.write(wget_script_content)\n",
    "    \n",
    "import os, subprocess\n",
    "os.chmod(script_path, 0o750)\n",
    "download_dir = os.path.dirname(script_path)\n",
    "subprocess.check_output(\"{}\".format(script_path), cwd=download_dir)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "… and the files will be downloaded to a temporary directory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(download_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are doing batch searching and things are running slow, you might be able to achieve a considerable speed up by sending the following argument to the search call:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ctx.search(ignore_facet_check=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This cuts out an extra call that typically takes 2 seconds to return a response. Note that it may mean some of the functionality is affected (such as being able to view the available facets and access the hit count) so use this feature with care."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also dictate how the search batches up its requests with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ctx.search(batch_size=250)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ``batch_size`` argument does not affect the final result but may affect the speed of the response. The batch size can also be set as a default in the ``pyesgf.search.consts`` module."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}