AutoRclone_(break_750Gb_Limit)v2.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "AutoRclone (break 750Gb_Limit)v2.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B5ssDEUtf2RD"
      },
      "source": [
        "# <center><a href=\"https://t.me/EduTechTainMent\"><img src='https://i.imgur.com/CLg6blO.png' height=\"70\" alt=\"Telegram Group\"/><a href=\"https://imgur.com/AjJvoLp\"><a href=\"https://edutechtainment.com\"><img src=\"https://i.imgur.com/AjJvoLp.png\" title=\"Over Every Possessor of Knowledge, There is (Some) One (Else) More Knowledgeable\" alt=\"EduTechTainMent\"/></a><a href=\"https://github.com/Waga43\"><img src='https://upload.wikimedia.org/wikipedia/commons/thumb/e/eb/Ei-sc-github.svg/768px-Ei-sc-github.svg.png' height=\"80\" alt=\"Github\"/></a><img src=\"https://hitcounter.pythonanywhere.com/count/tag.svg?url=https%3A%2F%2Fgithub.com%2Fbiplobsd%2FOneClickRun\" alt=\"Hits\" align=\"right\"></center>\n",
        "\n",
        "<b><center>Credits: </b></center>\n",
        " <emph><center>1. `Eric Li- Colab Notebook` </emph></center>\n",
        "<emph><center>2. `EduTechTainMent: Translation ONLY with some MINOR modifications `</center></emph>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Cqy3vXXOMI06"
      },
      "source": [
        "### <font size=5px color=\"red\">✦ *Google Colab Automatically disconnect after 90 minutes. We need a little tweak to by pass this \"limitation\":</font>\n",
        "<p><font size=3px > Automatically run the code every 60 minutes to refresh the 90-minute disconnect limit. Open developer-settings (in your browser). Windows Quick method: Click <i>\"Ctrl + Shift + I\"</i> and press console to enter the code given below. For Mac: Press <i>\"Option + Command + I\"</i>. Copy the following code below and paste it in the browser console! ! <b> Don't close the browser to avoid failure! </b>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3Z8cK8h2Avbv"
      },
      "source": [
        "<code>function ClickConnect(){\n",
        "console.log(\"Working\"); \n",
        "document.querySelector(\"colab-connect-button\").click() \n",
        "}setInterval(ClickConnect,6000)</code>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jtClEMAMLVHw",
        "cellView": "form"
      },
      "source": [
        "#@markdown <h3>← Run after entering the above code in your browser console (to prevent disconnection). </h>\n",
        "\n",
        "\n",
        "import IPython\n",
        "from google.colab import output\n",
        "\n",
        "display(IPython.display.Javascript('''\n",
        " function ClickConnect(){\n",
        "   btn = document.querySelector(\"colab-connect-button\")\n",
        "   if (btn != null){\n",
        "     console.log(\"Click colab-connect-button\"); \n",
        "     btn.click() \n",
        "     }\n",
        "   \n",
        "   btn = document.getElementById('ok')\n",
        "   if (btn != null){\n",
        "     console.log(\"Click reconnect\"); \n",
        "     btn.click() \n",
        "     }\n",
        "  }\n",
        "  \n",
        "setInterval(ClickConnect,60000)\n",
        "'''))\n",
        "\n",
        "print(\"Done.\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nnK31oX-jC5p"
      },
      "source": [
        "### <img src='https://rclone.org/img/rclone-32x32.png' height=\"46\" alt=\"rclone\"/>__Autorclone__\n",
        "<font size=3px><b> ✧  Google breaks the 750GB limit </h6>\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "W61wx2ZLjC52"
      },
      "source": [
        "#@markdown <h3><b>←</b> First step. Install rclone configuration environment</h3>\n",
        "\n",
        "!curl https://rclone.org/install.sh | sudo bash\n",
        "###\n",
        "%cd /content/\n",
        "!git clone https://github.com/totalleecher/AutoRclone.git\n",
        "###\n",
        "%cd /content/AutoRclone\n",
        "!sudo pip3 install -r requirements.txt\n",
        "\n",
        "from IPython.display import HTML, clear_output\n",
        "clear_output()\n",
        "print(\"Successfully Installed\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "XbcSRo0RjC6P"
      },
      "source": [
        "#@markdown <h3><b>←</b>The second step. Start Drive Api & download credentials.json to this machine</h3>\n",
        "!python3 gen_sa_accounts.py --quick-setup -1\n",
        "\n",
        "from IPython.display import Image, display\n",
        "display(Image(url = \" https://raw.githubusercontent.com/totalleecher/Codemaster/master/Screenshot_2020-03-24%20Python%20Quickstart%20Google%20Drive%20API%20Google%20Developers.png\"))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "N8fETzRpjC6s"
      },
      "source": [
        "#@markdown <h3><b>←</b>Third step. Upload the \"credentials.json\" just downloaded</h3>\n",
        "\n",
        "from google.colab import files\n",
        "uploaded = files.upload()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "LMoXkToAjC69"
      },
      "source": [
        "#@markdown <h3><b>←</b>The fourth step. Generate Service Accounts (SA) on existing projects \"If you already have SA, you can skip\"</h3>\n",
        "\n",
        "!python3 gen_sa_accounts.py --quick-setup -1"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vGIC3QKIIpfb"
      },
      "source": [
        "<font size=\"4\" color=red><b>If the fourth step has been wrong, you may not have any project. Click here to open <a href=\"https://console.developers.google.com/\" target=\"_blank\">Google Cloud Platform</a> ，In the upper left corner  GoogleAPIs Side press \"Select a project\" Press in the upper right corner of the pop-up window \"NEW PROJECT\". Then press \"CREATE\". Run the fourth step back。</b></font>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eplOxOmrjC7R"
      },
      "source": [
        "### Method1 (Put Services Acc Add to **Team Drive**)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "_-MmxPYFjC7W"
      },
      "source": [
        "#@markdown <h3><b>← Run. Put all SA Add to Team Drive</b></h3>\n",
        "#@markdown </h5>Team Drive ID (TDid) = https://drive.google.com/drive/folders/All the letters after. Please be patient just incase SA will consume a little time </b></h5> \n",
        "import os\n",
        "TDid = \"0APYAWm1rKPWNUk9PVA\" #@param {type:\"string\"}\n",
        "!python3 add_to_team_drive.py -d \"$TDid\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XCiA0yRMxM5T"
      },
      "source": [
        "### **<font size=5px color=bule><<<<<<<<<<<<<<<<<<<< Ready to start copying >>>>>>>>>>>>>>>>>>>>**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "ItewOXbSjC7r"
      },
      "source": [
        "#@markdown <p>- <b>character ID （Shared resources） And the exported folder ID （一It's usually your own team）</p>\n",
        "#@markdown <p>- Note: Your source file ID *** * (* `src` *) * *** folder or file to be shared publicly* ***</p>\n",
        "\n",
        "#@markdown <b>- The shared folder ID is as shown in the figure below: the characters behind folder /: just the characters behind the stickers</b>\n",
        "#@markdown ![](https://yuju.pw/y/qNvy.png)\n",
        "\n",
        "#@title <h3><b>←</b> Run to clone all folders + files from <code>src</code> to <code>dest</code>\n",
        "\n",
        "source = \"0AN4XtN1OGhoYUk9PVA\" #@param {type:\"string\"}\n",
        "destination = \"18s4CaTc-3ZjhufVEh-5EkRqxcgeGURkq\" #@param {type:\"string\"}\n",
        "sa = \"2\" #@param {type:\"string\"}\n",
        "\n",
        "!python3 rclone_sa_magic.py -s \"$source\" -d \"$destination\" -b \"$sa\" -e 600\n",
        "\n",
        "#@markdown <b>- SA starts at default 1. If you have copied 7.5T, then you should put 11 SA 750GB in 24 hours."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TR6XGxkt17-f",
        "cellView": "form"
      },
      "source": [
        "#@markdown <br><center><img src='https://upload.wikimedia.org/wikipedia/commons/thumb/d/da/Google_Drive_logo.png/600px-Google_Drive_logo.png' height=\"50\" alt=\"Gdrive-logo\"/></center>\n",
        "#@markdown <center><h3>Mount Gdrive to <b>/content/drive</b></h3></center><br>\n",
        "MODE = \"MOUNT\" #@param [\"MOUNT\", \"UNMOUNT\"]\n",
        "Dir_Path = \"\"\n",
        "#Mount your Gdrive! \n",
        "from google.colab import drive\n",
        "drive.mount._DEBUG = False\n",
        "if MODE == \"MOUNT\":\n",
        "  drive.mount('/content/drive', force_remount=True)\n",
        "elif MODE == \"UNMOUNT\":\n",
        "  try:\n",
        "    drive.flush_and_unmount()\n",
        "  except ValueError:\n",
        "    pass\n",
        "  get_ipython().system_raw(\"rm -rf /root/.config/Google/DriveFS\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NKQKbbO5jC8f"
      },
      "source": [
        "### Additional features (check resource size and number) ###"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "XWWAn0uujC8j"
      },
      "source": [
        "#@markdown <h4><b>←</b> Source file size</h3> \n",
        "!rclone --config rclone.conf size --disable ListR src001:"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "LsChZJ2FjC8y"
      },
      "source": [
        "#@markdown <h4><b>←</b> Target file size after dumping </h3> \n",
        "!rclone --config rclone.conf size --disable ListR src001:"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "bNcxDO0AMSkk"
      },
      "source": [
        "#@markdown <h4><b>←</b> Deduplicated target file after transfer </h3> \n",
        "!rclone --config rclone.conf dedupe dst001:"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z-0L1Mousoom"
      },
      "source": [
        "###<font size=4 color=\"red\"><b>Deduplicated target file after transfer <a href=\"https://console.developers.google.com/\" target=\"_blank\">Google Cloud Platform.</a> Copy the project id and paste it into the following project id. Import Services Account into the configuration environment and start copying </b></font>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "waf_p3KUre00",
        "cellView": "form"
      },
      "source": [
        "#@markdown <h3>←Enter the project id as shown below and run\n",
        "\n",
        "#@markdown ![](https://i.imgur.com/6kdw2u9.jpg)\n",
        "\n",
        "\n",
        "#@markdown <h4><b>Just paste in project id:</b>\n",
        "\n",
        "projectid = \"\" #@param {type:\"string\"}\n",
        "!python3 gen_sa_accounts.py --download-keys \"$projectid\""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}