esgf_vars_downloaded.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(C) Crown Copyright, Met Office. All rights reserved."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## esgf_vars_downloaded.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook looks at the number of downloads of CMIP6.HighResMIP variables that have been downloaded from CEDA's ESGF node via HTTP between 25th March 2019 and 14th May 2020. The variables have been sorted by frequency and table name. The number shows the number of times that unique datasets (e.g. a unique combination of institute, model, experiment, variant label, table and variable) have been downloaded from a unique IP address. \n",
    "\n",
    "PRIMAVERA was given access to access logs of the THREDDS Tomcat server from CEDA's ESGF node. All of the PRIMAVERA data has been published to this ESGF server. Data can be downloaded over the HTTP protocol from THREDDS or via Globus and so these logs do not show all of the data that has been downloaded, although some of the Globus downloads will have been replication of the data to other ESGF nodes and so may not be representative of actual usage of the data. The data may have been replicated to other ESGF nodes and so these logs only show downloads from CEDA’s ESGF node and not the total global downloads. \n",
    "\n",
    "The logs had been anonymised by replacing the IP address in them with a hash of the IP address. Some institutes will operate a web proxy so that all users at that institute will appear to come from the same IP address. Because some datasets from some institutes have a single variable spread over several files, a download has been counted as a download of that variable from that IP address hash. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Last run 2020-06-30 11:51:38.314441\n"
     ]
    }
   ],
   "source": [
    "import datetime\n",
    "print(f'Last run {datetime.datetime.utcnow()}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import OrderedDict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The path to the Tomcar log file\n",
    "LOGFILE = 'download-logs'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "class VariableRequests:\n",
    "    \"\"\"\n",
    "    A class to represent the variable requests and to keep track of how many\n",
    "    times each of them has been requested\n",
    "    \"\"\"\n",
    "    def __init__(self):\n",
    "        \"\"\"Create an empty dict of variable requests\"\"\"\n",
    "        self._vreqs = {}\n",
    "\n",
    "    def increment_vreq(self, vreq):\n",
    "        \"\"\"\n",
    "        Increment the retrieval count for `vreq`, adding it to the list if\n",
    "        it doesn't alreday exist.\n",
    "\n",
    "        :param str vreq: the variable request code\n",
    "        \"\"\"\n",
    "        if vreq not in self._vreqs:\n",
    "            self._vreqs[vreq] = 1\n",
    "        else:\n",
    "            self._vreqs[vreq] += 1\n",
    "\n",
    "    def get_vreqs(self, order_by_count=False):\n",
    "        \"\"\"\n",
    "        Get the list and count of variable requests. The default order of\n",
    "        requests is frequency, then table and finally variable name. They\n",
    "        can alternatively be returned in decreasing count order.\n",
    "\n",
    "        :param bool order_by_count: if True then return in decreasing order\n",
    "            of count.\n",
    "        :returns: the list and count of variable requests.\n",
    "        :rtype: str\n",
    "        \"\"\"\n",
    "        return_strings = []\n",
    "        if order_by_count:\n",
    "            ordered = OrderedDict(sorted(self._vreqs.items(),\n",
    "                                         key=lambda x: x[1],\n",
    "                                         reverse=True))\n",
    "        else:\n",
    "            ordered = OrderedDict(\n",
    "                sorted(self._vreqs.items(),\n",
    "                       # sort order is frequency then table name then variable\n",
    "                       key=lambda x: (_guess_frequency(x[0]),\n",
    "                                      x[0].split('_')[1],\n",
    "                                      x[0].split('_')[0]))\n",
    "            )\n",
    "        for vr in ordered:\n",
    "            return_strings.append('{:<25} {:3}'.format(vr, self._vreqs[vr]))\n",
    "\n",
    "        return '\\n'.join(return_strings)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def _guess_frequency(table_name):\n",
    "    \"\"\"\n",
    "    Return an integer corresponding to the frequency of variables in the table.\n",
    "    Higher frequency data (starting at 1hr) has a lower priority.\n",
    "\n",
    "    :param str table_name: a string containing the table name.\n",
    "    :returns: an integer corresponding to the frequency of variables.\n",
    "    :rtype: int\n",
    "    :raises ValueError: if a valid frequency isn't found in the table name.\n",
    "    \"\"\"\n",
    "    frequencies = {\n",
    "        '1hr': 1,\n",
    "        '3hr': 2,\n",
    "        '6hr': 3,\n",
    "        'day': 4,\n",
    "        'mon': 5,\n",
    "        'fx': 6\n",
    "    }\n",
    "    for freq in frequencies:\n",
    "        if freq in table_name:\n",
    "            return frequencies[freq]\n",
    "\n",
    "    raise ValueError(f'No frequency found for table name {table_name}')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2216976 lines processed\n"
     ]
    }
   ],
   "source": [
    "dreqs = {}\n",
    "vreqs = VariableRequests()\n",
    "num_bad_request = 0\n",
    "num_other_requests = 0\n",
    "num_lines_done = 0\n",
    "\n",
    "with open(LOGFILE) as fh:\n",
    "    for line in fh:\n",
    "        num_lines_done += 1\n",
    "        cmpts = line.split()\n",
    "        ip_hash = cmpts[0]\n",
    "        url = cmpts[6]\n",
    "        status = cmpts[-2]\n",
    "\n",
    "        if status != '200':\n",
    "            # bad request so ignore and move to next\n",
    "            num_bad_request += 1\n",
    "            continue\n",
    "\n",
    "        if not url.startswith('/thredds/fileServer/esg_cmip6'):\n",
    "            # not a file retrieval so ignore and move to next\n",
    "            num_other_requests += 1\n",
    "            continue\n",
    "\n",
    "        url_parts = url.split('/')\n",
    "        # The data request code is in the form:\n",
    "        # institute_id/source_id/variant_label/table_name/cmor_name\n",
    "        dreq = '/'.join(url_parts[6:12])\n",
    "        # The variable request code is in the form:\n",
    "        # cmor_name_table_name\n",
    "        vreq = f'{url_parts[11]}_{url_parts[10]}'\n",
    "        if dreq not in dreqs:\n",
    "            # This data request hasn't been requested before\n",
    "            dreqs[dreq] = [ip_hash]\n",
    "            vreqs.increment_vreq(vreq)\n",
    "        elif ip_hash not in dreqs[dreq]:\n",
    "            # This IP address hasn't requested this data request before\n",
    "            dreqs[dreq].append(ip_hash)\n",
    "            vreqs.increment_vreq(vreq)\n",
    "\n",
    "print(f'{num_lines_done} lines processed')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look at the variables downloaded in frequency and table order"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pr_E1hr                    48\n",
      "prc_E1hr                   38\n",
      "clt_3hr                     7\n",
      "hfls_3hr                    4\n",
      "hfss_3hr                    6\n",
      "huss_3hr                   38\n",
      "mrsos_3hr                   1\n",
      "pr_3hr                    107\n",
      "prc_3hr                     7\n",
      "prsn_3hr                    5\n",
      "ps_3hr                      3\n",
      "rlds_3hr                   13\n",
      "rldscs_3hr                  1\n",
      "rlus_3hr                   14\n",
      "rsds_3hr                   13\n",
      "rsdscs_3hr                 12\n",
      "rsus_3hr                   12\n",
      "rsuscs_3hr                 14\n",
      "tas_3hr                    77\n",
      "tos_3hr                     5\n",
      "tslsi_3hr                   4\n",
      "uas_3hr                    63\n",
      "vas_3hr                    60\n",
      "psl_CF3hr                   3\n",
      "prcsh_E3hr                  2\n",
      "prw_E3hr                   14\n",
      "psl_E3hr                   36\n",
      "rlut_E3hr                   1\n",
      "rlutcs_E3hr                 1\n",
      "rsut_E3hr                   1\n",
      "rsutcs_E3hr                 9\n",
      "hus_E3hrPt                  7\n",
      "ta_E3hrPt                   7\n",
      "ua_E3hrPt                  11\n",
      "va_E3hrPt                   9\n",
      "psl_6hrPlev                40\n",
      "wap_6hrPlev                 2\n",
      "hus_6hrPlevPt              25\n",
      "huss_6hrPlevPt              2\n",
      "psl_6hrPlevPt              95\n",
      "rv850_6hrPlevPt             5\n",
      "sfcWind_6hrPlevPt           2\n",
      "ta_6hrPlevPt               63\n",
      "tas_6hrPlevPt              72\n",
      "ts_6hrPlevPt                3\n",
      "ua_6hrPlevPt               79\n",
      "uas_6hrPlevPt              90\n",
      "va_6hrPlevPt               84\n",
      "vas_6hrPlevPt              94\n",
      "wbptemp_6hrPlevPt          16\n",
      "zg_6hrPlevPt               31\n",
      "zg500_6hrPlevPt             7\n",
      "albisccp_CFday              9\n",
      "ccb_CFday                   9\n",
      "cct_CFday                   9\n",
      "cl_CFday                   15\n",
      "clcalipso_CFday             3\n",
      "clhcalipso_CFday            3\n",
      "cli_CFday                   9\n",
      "clisccp_CFday               4\n",
      "clivi_CFday                 4\n",
      "cllcalipso_CFday            2\n",
      "clmcalipso_CFday            3\n",
      "cltcalipso_CFday            7\n",
      "cltisccp_CFday             10\n",
      "clw_CFday                  12\n",
      "clwvi_CFday                 4\n",
      "hur_CFday                   3\n",
      "hus_CFday                   4\n",
      "mc_CFday                    5\n",
      "pctisccp_CFday              9\n",
      "pfull_CFday                18\n",
      "phalf_CFday                16\n",
      "ps_CFday                   40\n",
      "rldscs_CFday               10\n",
      "rlutcs_CFday               15\n",
      "rsdscs_CFday               15\n",
      "rsdt_CFday                 15\n",
      "rsuscs_CFday               14\n",
      "rsut_CFday                 18\n",
      "rsutcs_CFday               17\n",
      "ta_CFday                    5\n",
      "ta700_CFday                19\n",
      "ua_CFday                    5\n",
      "va_CFday                    3\n",
      "wap_CFday                   3\n",
      "wap500_CFday                9\n",
      "zg_CFday                    5\n",
      "parasolRefl_Eday            6\n",
      "rivo_Eday                  10\n",
      "ta_Eday                     1\n",
      "tauu_Eday                  18\n",
      "tauv_Eday                  16\n",
      "ts_Eday                    15\n",
      "ua_Eday                     4\n",
      "va_Eday                     4\n",
      "epfz_EdayZ                  4\n",
      "utendnogw_EdayZ             1\n",
      "utendogw_EdayZ              2\n",
      "vtem_EdayZ                  4\n",
      "wtem_EdayZ                  4\n",
      "zg_EdayZ                    2\n",
      "omldamax_Oday               5\n",
      "sos_Oday                    9\n",
      "tos_Oday                   54\n",
      "tossq_Oday                  7\n",
      "siconc_SIday               26\n",
      "siconca_SIday               1\n",
      "sisnthick_SIday             6\n",
      "sitemptop_SIday             7\n",
      "sithick_SIday              20\n",
      "siu_SIday                   9\n",
      "siv_SIday                  10\n",
      "clt_day                    29\n",
      "hfls_day                   40\n",
      "hfss_day                   39\n",
      "hur_day                    42\n",
      "hurs_day                   48\n",
      "hursmax_day                12\n",
      "hursmin_day                10\n",
      "hus_day                   158\n",
      "huss_day                   98\n",
      "mrro_day                   43\n",
      "mrso_day                   13\n",
      "mrsos_day                  17\n",
      "pr_day                    545\n",
      "prc_day                    34\n",
      "prsn_day                   37\n",
      "psl_day                   155\n",
      "rlds_day                   55\n",
      "rlus_day                   54\n",
      "rlut_day                   44\n",
      "rsds_day                   81\n",
      "rsus_day                   47\n",
      "sfcWind_day                68\n",
      "sfcWindmax_day             20\n",
      "snc_day                     6\n",
      "snw_day                     9\n",
      "ta_day                    168\n",
      "tas_day                   235\n",
      "tasmax_day                218\n",
      "tasmin_day                203\n",
      "tslsi_day                  21\n",
      "ua_day                    163\n",
      "uas_day                   152\n",
      "va_day                    145\n",
      "vas_day                   139\n",
      "wap_day                    38\n",
      "zg_day                     96\n",
      "abs550aer_AERmon            1\n",
      "cltc_AERmon                 1\n",
      "cod_AERmon                  1\n",
      "lwp_AERmon                  1\n",
      "od550aer_AERmon             2\n",
      "od550dust_AERmon            1\n",
      "od550oa_AERmon              3\n",
      "od550so4_AERmon             1\n",
      "od550ss_AERmon              1\n",
      "ptp_AERmon                  7\n",
      "reffclwtop_AERmon          25\n",
      "rlutaf_AERmon               6\n",
      "rlutcsaf_AERmon             1\n",
      "rsutaf_AERmon               5\n",
      "rsutcsaf_AERmon             5\n",
      "tatp_AERmon                 1\n",
      "toz_AERmon                  5\n",
      "ttop_AERmon                 1\n",
      "ua_AERmon                   2\n",
      "va_AERmon                   1\n",
      "ztp_AERmon                  1\n",
      "ccb_Amon                   30\n",
      "cct_Amon                   28\n",
      "ch4global_Amon              4\n",
      "ci_Amon                    26\n",
      "cl_Amon                    61\n",
      "cli_Amon                   44\n",
      "clivi_Amon                 50\n",
      "clt_Amon                   82\n",
      "clw_Amon                   44\n",
      "clwvi_Amon                 47\n",
      "co2mass_Amon                6\n",
      "evspsbl_Amon              133\n",
      "hfls_Amon                 109\n",
      "hfss_Amon                 109\n",
      "hur_Amon                   91\n",
      "hurs_Amon                  85\n",
      "hus_Amon                  223\n",
      "huss_Amon                 144\n",
      "mc_Amon                     5\n",
      "n2oglobal_Amon              4\n",
      "pfull_Amon                 14\n",
      "phalf_Amon                 14\n",
      "pr_Amon                   716\n",
      "prc_Amon                   88\n",
      "prsn_Amon                  75\n",
      "prw_Amon                  121\n",
      "ps_Amon                   214\n",
      "psl_Amon                  342\n",
      "rlds_Amon                 119\n",
      "rldscs_Amon                70\n",
      "rlus_Amon                  98\n",
      "rlut_Amon                  94\n",
      "rlutcs_Amon                59\n",
      "rsds_Amon                 119\n",
      "rsdscs_Amon                69\n",
      "rsdt_Amon                 108\n",
      "rsus_Amon                  97\n",
      "rsuscs_Amon                70\n",
      "rsut_Amon                  96\n",
      "rsutcs_Amon                58\n",
      "rtmt_Amon                   8\n",
      "sbl_Amon                   29\n",
      "sci_Amon                   21\n",
      "sfcWind_Amon               80\n",
      "ta_Amon                   303\n",
      "tas_Amon                  627\n",
      "tasmax_Amon               104\n",
      "tasmin_Amon               105\n",
      "tauu_Amon                 111\n",
      "tauv_Amon                 109\n",
      "ts_Amon                   294\n",
      "ua_Amon                   317\n",
      "uas_Amon                  291\n",
      "va_Amon                   289\n",
      "vas_Amon                  281\n",
      "wap_Amon                  136\n",
      "zg_Amon                   259\n",
      "albisccp_CFmon             22\n",
      "clis_CFmon                  1\n",
      "clisccp_CFmon               1\n",
      "cllcalipso_CFmon            1\n",
      "cltcalipso_CFmon           22\n",
      "cltisccp_CFmon             24\n",
      "hur_CFmon                  27\n",
      "hus_CFmon                  39\n",
      "pctisccp_CFmon             23\n",
      "rld_CFmon                  26\n",
      "rldcs_CFmon                23\n",
      "rlu_CFmon                  23\n",
      "rlucs_CFmon                23\n",
      "rlucs4co2_CFmon             1\n",
      "rlut4co2_CFmon              5\n",
      "rsd_CFmon                  23\n",
      "rsdcs_CFmon                21\n",
      "rsdcs4co2_CFmon             2\n",
      "rsu_CFmon                  23\n",
      "rsu4co2_CFmon               2\n",
      "rsucs_CFmon                24\n",
      "rsut4co2_CFmon              4\n",
      "ta_CFmon                   37\n",
      "tnhus_CFmon                 2\n",
      "tnhusa_CFmon                1\n",
      "tnhusc_CFmon                1\n",
      "tnhusmp_CFmon               1\n",
      "tnhusscpbl_CFmon            1\n",
      "tntscpbl_CFmon              2\n",
      "evspsblpot_Emon             3\n",
      "hcont300_Emon               1\n",
      "hus_Emon                    9\n",
      "intuadse_Emon               2\n",
      "intuaw_Emon                 1\n",
      "intvadse_Emon               3\n",
      "intvaw_Emon                 1\n",
      "mrsol_Emon                  2\n",
      "mrtws_Emon                  4\n",
      "nep_Emon                    3\n",
      "parasolRefl_Emon           20\n",
      "sfcWindmax_Emon             2\n",
      "t20d_Emon                   1\n",
      "ta_Emon                     8\n",
      "ua_Emon                     9\n",
      "uqint_Emon                 25\n",
      "va_Emon                     9\n",
      "vqint_Emon                 22\n",
      "wtd_Emon                    1\n",
      "zg_Emon                     8\n",
      "epfz_EmonZ                 17\n",
      "vtem_EmonZ                 16\n",
      "wtem_EmonZ                 17\n",
      "agesno_LImon                1\n",
      "hfdsn_LImon                 1\n",
      "lwsnl_LImon                 3\n",
      "sbl_LImon                   3\n",
      "snc_LImon                  31\n",
      "snd_LImon                  66\n",
      "snm_LImon                  48\n",
      "snw_LImon                  53\n",
      "tsn_LImon                  34\n",
      "baresoilFrac_Lmon           2\n",
      "c4PftFrac_Lmon              1\n",
      "cropFrac_Lmon               2\n",
      "evspsblsoi_Lmon            21\n",
      "evspsblveg_Lmon            34\n",
      "gpp_Lmon                   37\n",
      "grassFrac_Lmon              1\n",
      "lai_Lmon                   37\n",
      "mrfso_Lmon                  4\n",
      "mrro_Lmon                 130\n",
      "mrros_Lmon                 78\n",
      "mrso_Lmon                  92\n",
      "mrsos_Lmon                 35\n",
      "npp_Lmon                   43\n",
      "ra_Lmon                    34\n",
      "rh_Lmon                    45\n",
      "tran_Lmon                  10\n",
      "treeFrac_Lmon               1\n",
      "tsl_Lmon                   95\n",
      "agessc_Omon                 1\n",
      "bigthetao_Omon              2\n",
      "bigthetaoga_Omon            1\n",
      "hfbasinpmadv_Omon           1\n",
      "hfds_Omon                  30\n",
      "hfx_Omon                    1\n",
      "hfy_Omon                    2\n",
      "htovovrt_Omon               1\n",
      "masso_Omon                  1\n",
      "mlotst_Omon                28\n",
      "mlotstsq_Omon               4\n",
      "msftmz_Omon                 2\n",
      "obvfsq_Omon                 1\n",
      "pbo_Omon                    3\n",
      "rsdo_Omon                   1\n",
      "rsntds_Omon                 8\n",
      "sltovgyre_Omon              2\n",
      "sltovovrt_Omon              1\n",
      "so_Omon                    28\n",
      "soga_Omon                   1\n",
      "sos_Omon                   20\n",
      "tauuo_Omon                 27\n",
      "tauvo_Omon                 23\n",
      "thetao_Omon                55\n",
      "thetaoga_Omon              10\n",
      "thkcello_Omon              55\n",
      "tos_Omon                   82\n",
      "tossq_Omon                  6\n",
      "umo_Omon                    4\n",
      "uo_Omon                    26\n",
      "vmo_Omon                    4\n",
      "vo_Omon                    27\n",
      "volo_Omon                   2\n",
      "wfo_Omon                    5\n",
      "wmo_Omon                    7\n",
      "wo_Omon                    19\n",
      "zfullo_Omon                 2\n",
      "zos_Omon                   25\n",
      "zossq_Omon                  3\n",
      "zostoga_Omon                2\n",
      "siage_SImon                 1\n",
      "siarean_SImon               8\n",
      "siareas_SImon               7\n",
      "sicompstren_SImon           1\n",
      "siconc_SImon               47\n",
      "siconca_SImon               7\n",
      "sidconcdyn_SImon            4\n",
      "sidconcth_SImon             4\n",
      "sidmassdyn_SImon            5\n",
      "sidmassevapsubl_SImon       1\n",
      "sidmasslat_SImon            1\n",
      "sidmassth_SImon             4\n",
      "sidmasstranx_SImon          4\n",
      "sidmasstrany_SImon          5\n",
      "siextentn_SImon            10\n",
      "siextents_SImon            10\n",
      "sifb_SImon                  4\n",
      "siflcondbot_SImon           4\n",
      "siflcondtop_SImon           4\n",
      "siflfwbot_SImon             4\n",
      "siflswdtop_SImon            2\n",
      "sihc_SImon                  5\n",
      "simass_SImon                8\n",
      "simassacrossline_SImon      4\n",
      "sisali_SImon                1\n",
      "sisaltmass_SImon            4\n",
      "sisnconc_SImon              5\n",
      "sisnhc_SImon                4\n",
      "sisnmass_SImon             11\n",
      "sisnthick_SImon            22\n",
      "sispeed_SImon               7\n",
      "sistrxdtop_SImon           22\n",
      "sistrxubot_SImon            4\n",
      "sistrydtop_SImon           26\n",
      "sistryubot_SImon            4\n",
      "sitemptop_SImon            19\n",
      "sithick_SImon              29\n",
      "sitimefrac_SImon            8\n",
      "siu_SImon                  23\n",
      "siv_SImon                  24\n",
      "sivol_SImon                28\n",
      "sivoln_SImon                4\n",
      "sivols_SImon                7\n",
      "sndmassdyn_SImon            4\n",
      "sndmasssnf_SImon            4\n"
     ]
    }
   ],
   "source": [
    "print(vreqs.get_vreqs())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look at the variables downloaded in popularity order"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pr_Amon                   716\n",
      "tas_Amon                  627\n",
      "pr_day                    545\n",
      "psl_Amon                  342\n",
      "ua_Amon                   317\n",
      "ta_Amon                   303\n",
      "ts_Amon                   294\n",
      "uas_Amon                  291\n",
      "va_Amon                   289\n",
      "vas_Amon                  281\n",
      "zg_Amon                   259\n",
      "tas_day                   235\n",
      "hus_Amon                  223\n",
      "tasmax_day                218\n",
      "ps_Amon                   214\n",
      "tasmin_day                203\n",
      "ta_day                    168\n",
      "ua_day                    163\n",
      "hus_day                   158\n",
      "psl_day                   155\n",
      "uas_day                   152\n",
      "va_day                    145\n",
      "huss_Amon                 144\n",
      "vas_day                   139\n",
      "wap_Amon                  136\n",
      "evspsbl_Amon              133\n",
      "mrro_Lmon                 130\n",
      "prw_Amon                  121\n",
      "rsds_Amon                 119\n",
      "rlds_Amon                 119\n",
      "tauu_Amon                 111\n",
      "hfls_Amon                 109\n",
      "hfss_Amon                 109\n",
      "tauv_Amon                 109\n",
      "rsdt_Amon                 108\n",
      "pr_3hr                    107\n",
      "tasmin_Amon               105\n",
      "tasmax_Amon               104\n",
      "rlus_Amon                  98\n",
      "huss_day                   98\n",
      "rsus_Amon                  97\n",
      "rsut_Amon                  96\n",
      "zg_day                     96\n",
      "tsl_Lmon                   95\n",
      "psl_6hrPlevPt              95\n",
      "rlut_Amon                  94\n",
      "vas_6hrPlevPt              94\n",
      "mrso_Lmon                  92\n",
      "hur_Amon                   91\n",
      "uas_6hrPlevPt              90\n",
      "prc_Amon                   88\n",
      "hurs_Amon                  85\n",
      "va_6hrPlevPt               84\n",
      "clt_Amon                   82\n",
      "tos_Omon                   82\n",
      "rsds_day                   81\n",
      "sfcWind_Amon               80\n",
      "ua_6hrPlevPt               79\n",
      "mrros_Lmon                 78\n",
      "tas_3hr                    77\n",
      "prsn_Amon                  75\n",
      "tas_6hrPlevPt              72\n",
      "rldscs_Amon                70\n",
      "rsuscs_Amon                70\n",
      "rsdscs_Amon                69\n",
      "sfcWind_day                68\n",
      "snd_LImon                  66\n",
      "ta_6hrPlevPt               63\n",
      "uas_3hr                    63\n",
      "cl_Amon                    61\n",
      "vas_3hr                    60\n",
      "rlutcs_Amon                59\n",
      "rsutcs_Amon                58\n",
      "thetao_Omon                55\n",
      "rlds_day                   55\n",
      "thkcello_Omon              55\n",
      "tos_Oday                   54\n",
      "rlus_day                   54\n",
      "snw_LImon                  53\n",
      "clivi_Amon                 50\n",
      "pr_E1hr                    48\n",
      "hurs_day                   48\n",
      "snm_LImon                  48\n",
      "clwvi_Amon                 47\n",
      "rsus_day                   47\n",
      "siconc_SImon               47\n",
      "rh_Lmon                    45\n",
      "clw_Amon                   44\n",
      "rlut_day                   44\n",
      "cli_Amon                   44\n",
      "npp_Lmon                   43\n",
      "mrro_day                   43\n",
      "hur_day                    42\n",
      "psl_6hrPlev                40\n",
      "hfls_day                   40\n",
      "ps_CFday                   40\n",
      "hfss_day                   39\n",
      "hus_CFmon                  39\n",
      "prc_E1hr                   38\n",
      "huss_3hr                   38\n",
      "wap_day                    38\n",
      "ta_CFmon                   37\n",
      "gpp_Lmon                   37\n",
      "lai_Lmon                   37\n",
      "prsn_day                   37\n",
      "psl_E3hr                   36\n",
      "mrsos_Lmon                 35\n",
      "prc_day                    34\n",
      "ra_Lmon                    34\n",
      "evspsblveg_Lmon            34\n",
      "tsn_LImon                  34\n",
      "zg_6hrPlevPt               31\n",
      "snc_LImon                  31\n",
      "hfds_Omon                  30\n",
      "ccb_Amon                   30\n",
      "clt_day                    29\n",
      "sbl_Amon                   29\n",
      "sithick_SImon              29\n",
      "cct_Amon                   28\n",
      "mlotst_Omon                28\n",
      "so_Omon                    28\n",
      "sivol_SImon                28\n",
      "hur_CFmon                  27\n",
      "vo_Omon                    27\n",
      "tauuo_Omon                 27\n",
      "ci_Amon                    26\n",
      "siconc_SIday               26\n",
      "uo_Omon                    26\n",
      "rld_CFmon                  26\n",
      "sistrydtop_SImon           26\n",
      "hus_6hrPlevPt              25\n",
      "zos_Omon                   25\n",
      "uqint_Emon                 25\n",
      "reffclwtop_AERmon          25\n",
      "siv_SImon                  24\n",
      "rsucs_CFmon                24\n",
      "cltisccp_CFmon             24\n",
      "rlu_CFmon                  23\n",
      "tauvo_Omon                 23\n",
      "rsu_CFmon                  23\n",
      "pctisccp_CFmon             23\n",
      "rldcs_CFmon                23\n",
      "rlucs_CFmon                23\n",
      "rsd_CFmon                  23\n",
      "siu_SImon                  23\n",
      "vqint_Emon                 22\n",
      "cltcalipso_CFmon           22\n",
      "albisccp_CFmon             22\n",
      "sisnthick_SImon            22\n",
      "sistrxdtop_SImon           22\n",
      "evspsblsoi_Lmon            21\n",
      "sci_Amon                   21\n",
      "tslsi_day                  21\n",
      "rsdcs_CFmon                21\n",
      "sos_Omon                   20\n",
      "sfcWindmax_day             20\n",
      "sithick_SIday              20\n",
      "parasolRefl_Emon           20\n",
      "wo_Omon                    19\n",
      "ta700_CFday                19\n",
      "sitemptop_SImon            19\n",
      "tauu_Eday                  18\n",
      "pfull_CFday                18\n",
      "rsut_CFday                 18\n",
      "rsutcs_CFday               17\n",
      "mrsos_day                  17\n",
      "epfz_EmonZ                 17\n",
      "wtem_EmonZ                 17\n",
      "wbptemp_6hrPlevPt          16\n",
      "phalf_CFday                16\n",
      "tauv_Eday                  16\n",
      "vtem_EmonZ                 16\n",
      "rsdt_CFday                 15\n",
      "cl_CFday                   15\n",
      "ts_Eday                    15\n",
      "rlutcs_CFday               15\n",
      "rsdscs_CFday               15\n",
      "rlus_3hr                   14\n",
      "rsuscs_CFday               14\n",
      "rsuscs_3hr                 14\n",
      "prw_E3hr                   14\n",
      "phalf_Amon                 14\n",
      "pfull_Amon                 14\n",
      "rlds_3hr                   13\n",
      "rsds_3hr                   13\n",
      "mrso_day                   13\n",
      "rsdscs_3hr                 12\n",
      "clw_CFday                  12\n",
      "hursmax_day                12\n",
      "rsus_3hr                   12\n",
      "ua_E3hrPt                  11\n",
      "sisnmass_SImon             11\n",
      "tran_Lmon                  10\n",
      "siextents_SImon            10\n",
      "rldscs_CFday               10\n",
      "siextentn_SImon            10\n",
      "hursmin_day                10\n",
      "siv_SIday                  10\n",
      "cltisccp_CFday             10\n",
      "rivo_Eday                  10\n",
      "thetaoga_Omon              10\n",
      "wap500_CFday                9\n",
      "albisccp_CFday              9\n",
      "hus_Emon                    9\n",
      "pctisccp_CFday              9\n",
      "cli_CFday                   9\n",
      "cct_CFday                   9\n",
      "ccb_CFday                   9\n",
      "siu_SIday                   9\n",
      "snw_day                     9\n",
      "rsutcs_E3hr                 9\n",
      "sos_Oday                    9\n",
      "va_E3hrPt                   9\n",
      "va_Emon                     9\n",
      "ua_Emon                     9\n",
      "rtmt_Amon                   8\n",
      "siarean_SImon               8\n",
      "sitimefrac_SImon            8\n",
      "simass_SImon                8\n",
      "ta_Emon                     8\n",
      "zg_Emon                     8\n",
      "rsntds_Omon                 8\n",
      "ta_E3hrPt                   7\n",
      "clt_3hr                     7\n",
      "hus_E3hrPt                  7\n",
      "sispeed_SImon               7\n",
      "siconca_SImon               7\n",
      "zg500_6hrPlevPt             7\n",
      "tossq_Oday                  7\n",
      "sitemptop_SIday             7\n",
      "cltcalipso_CFday            7\n",
      "ptp_AERmon                  7\n",
      "prc_3hr                     7\n",
      "siareas_SImon               7\n",
      "sivols_SImon                7\n",
      "wmo_Omon                    7\n",
      "hfss_3hr                    6\n",
      "co2mass_Amon                6\n",
      "rlutaf_AERmon               6\n",
      "parasolRefl_Eday            6\n",
      "snc_day                     6\n",
      "sisnthick_SIday             6\n",
      "tossq_Omon                  6\n",
      "mc_CFday                    5\n",
      "rv850_6hrPlevPt             5\n",
      "ua_CFday                    5\n",
      "sidmasstrany_SImon          5\n",
      "prsn_3hr                    5\n",
      "zg_CFday                    5\n",
      "ta_CFday                    5\n",
      "omldamax_Oday               5\n",
      "rlut4co2_CFmon              5\n",
      "toz_AERmon                  5\n",
      "rsutaf_AERmon               5\n",
      "rsutcsaf_AERmon             5\n",
      "wfo_Omon                    5\n",
      "sihc_SImon                  5\n",
      "sidmassdyn_SImon            5\n",
      "sisnconc_SImon              5\n",
      "tos_3hr                     5\n",
      "mc_Amon                     5\n",
      "hfls_3hr                    4\n",
      "mrtws_Emon                  4\n",
      "hus_CFday                   4\n",
      "vmo_Omon                    4\n",
      "mlotstsq_Omon               4\n",
      "ua_Eday                     4\n",
      "clwvi_CFday                 4\n",
      "clivi_CFday                 4\n",
      "clisccp_CFday               4\n",
      "rsut4co2_CFmon              4\n",
      "tslsi_3hr                   4\n",
      "vtem_EdayZ                  4\n",
      "epfz_EdayZ                  4\n",
      "wtem_EdayZ                  4\n",
      "sistrxubot_SImon            4\n",
      "sistryubot_SImon            4\n",
      "sisaltmass_SImon            4\n",
      "sidconcth_SImon             4\n",
      "siflcondtop_SImon           4\n",
      "sisnhc_SImon                4\n",
      "sifb_SImon                  4\n",
      "simassacrossline_SImon      4\n",
      "sidconcdyn_SImon            4\n",
      "sndmasssnf_SImon            4\n",
      "sidmassth_SImon             4\n",
      "siflcondbot_SImon           4\n",
      "sndmassdyn_SImon            4\n",
      "sidmasstranx_SImon          4\n",
      "sivoln_SImon                4\n",
      "siflfwbot_SImon             4\n",
      "va_Eday                     4\n",
      "n2oglobal_Amon              4\n",
      "ch4global_Amon              4\n",
      "mrfso_Lmon                  4\n",
      "umo_Omon                    4\n",
      "od550oa_AERmon              3\n",
      "pbo_Omon                    3\n",
      "sbl_LImon                   3\n",
      "evspsblpot_Emon             3\n",
      "intvadse_Emon               3\n",
      "wap_CFday                   3\n",
      "va_CFday                    3\n",
      "hur_CFday                   3\n",
      "clcalipso_CFday             3\n",
      "clhcalipso_CFday            3\n",
      "clmcalipso_CFday            3\n",
      "nep_Emon                    3\n",
      "psl_CF3hr                   3\n",
      "zossq_Omon                  3\n",
      "lwsnl_LImon                 3\n",
      "ts_6hrPlevPt                3\n",
      "ps_3hr                      3\n",
      "od550aer_AERmon             2\n",
      "utendogw_EdayZ              2\n",
      "bigthetao_Omon              2\n",
      "tntscpbl_CFmon              2\n",
      "volo_Omon                   2\n",
      "tnhus_CFmon                 2\n",
      "intuadse_Emon               2\n",
      "cropFrac_Lmon               2\n",
      "rsu4co2_CFmon               2\n",
      "cllcalipso_CFday            2\n",
      "siflswdtop_SImon            2\n",
      "zg_EdayZ                    2\n",
      "sfcWindmax_Emon             2\n",
      "wap_6hrPlev                 2\n",
      "baresoilFrac_Lmon           2\n",
      "sfcWind_6hrPlevPt           2\n",
      "huss_6hrPlevPt              2\n",
      "ua_AERmon                   2\n",
      "rsdcs4co2_CFmon             2\n",
      "mrsol_Emon                  2\n",
      "hfy_Omon                    2\n",
      "sltovgyre_Omon              2\n",
      "zostoga_Omon                2\n",
      "msftmz_Omon                 2\n",
      "prcsh_E3hr                  2\n",
      "zfullo_Omon                 2\n",
      "rldscs_3hr                  1\n",
      "mrsos_3hr                   1\n",
      "tnhusc_CFmon                1\n",
      "rlucs4co2_CFmon             1\n",
      "bigthetaoga_Omon            1\n",
      "c4PftFrac_Lmon              1\n",
      "hfbasinpmadv_Omon           1\n",
      "clis_CFmon                  1\n",
      "obvfsq_Omon                 1\n",
      "masso_Omon                  1\n",
      "utendnogw_EdayZ             1\n",
      "rlutcs_E3hr                 1\n",
      "rsut_E3hr                   1\n",
      "siconca_SIday               1\n",
      "treeFrac_Lmon               1\n",
      "grassFrac_Lmon              1\n",
      "ta_Eday                     1\n",
      "hfx_Omon                    1\n",
      "soga_Omon                   1\n",
      "rsdo_Omon                   1\n",
      "t20d_Emon                   1\n",
      "hcont300_Emon               1\n",
      "tatp_AERmon                 1\n",
      "tnhusmp_CFmon               1\n",
      "tnhusscpbl_CFmon            1\n",
      "clisccp_CFmon               1\n",
      "od550ss_AERmon              1\n",
      "lwp_AERmon                  1\n",
      "ztp_AERmon                  1\n",
      "rlutcsaf_AERmon             1\n",
      "intvaw_Emon                 1\n",
      "wtd_Emon                    1\n",
      "od550dust_AERmon            1\n",
      "agesno_LImon                1\n",
      "hfdsn_LImon                 1\n",
      "sidmasslat_SImon            1\n",
      "abs550aer_AERmon            1\n",
      "ttop_AERmon                 1\n",
      "cod_AERmon                  1\n",
      "od550so4_AERmon             1\n",
      "intuaw_Emon                 1\n",
      "cltc_AERmon                 1\n",
      "tnhusa_CFmon                1\n",
      "cllcalipso_CFmon            1\n",
      "htovovrt_Omon               1\n",
      "sltovovrt_Omon              1\n",
      "rlut_E3hr                   1\n",
      "agessc_Omon                 1\n",
      "sicompstren_SImon           1\n",
      "siage_SImon                 1\n",
      "sidmassevapsubl_SImon       1\n",
      "sisali_SImon                1\n",
      "va_AERmon                   1\n"
     ]
    }
   ],
   "source": [
    "print(vreqs.get_vreqs(order_by_count=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}