Merge pull request #838 from sujee/html-rag-1

Html rag 1 -- Crawl a website / process HTML / run RAG queries

.gitignore

@@ -146,3 +146,7 @@ dmypy.json
 
 # Ignore artifacts directory created by the `make save-images` target
 artifacts/
+
+*.lock
+
+*.db

examples/notebooks/html-processing/.gitignore

@@ -0,0 +1,2 @@
+input*/
+output*/

examples/notebooks/html-processing/1_crawl_site.ipynb

@@ -0,0 +1,154 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Crawl a website \n",
+    "\n",
+    "We will use `DPK-Connector` module\n",
+    "\n",
+    "References\n",
+    "- https://github.com/IBM/data-prep-kit/tree/dev/data-connector-lib\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Step-1: Config"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from my_config import MY_CONFIG"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Step-2: Setup Directories"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "✅ Cleared  download directory\n"
+     ]
+    }
+   ],
+   "source": [
+    "import os, sys\n",
+    "import shutil\n",
+    "\n",
+    "if not os.path.exists(MY_CONFIG.INPUT_DIR ):\n",
+    "    shutil.os.makedirs(MY_CONFIG.INPUT_DIR, exist_ok=True)\n",
+    "\n",
+    "## clear input / output folder\n",
+    "shutil.rmtree(MY_CONFIG.INPUT_DIR, ignore_errors=True)\n",
+    "shutil.os.makedirs(MY_CONFIG.INPUT_DIR, exist_ok=True)\n",
+    "\n",
+    "print (\"✅ Cleared  download directory\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Step-3: Crawl the website\n",
+    "\n",
+    "We will use `dpk-connector` utility to download a some HTML pages from a site \n",
+    "\n",
+    "\n",
+    "**Parameters**\n",
+    "\n",
+    "For configuring the crawl, users need to specify the following parameters:\n",
+    "\n",
+    "| parameter:type | Description |\n",
+    "| --- | --- |\n",
+    "| urls:list | list of seed URLs (i.e., ['https://thealliance.ai'] or ['https://www.apache.org/projects','https://www.apache.org/foundation']). The list can include any number of valid URLS that are not configured to block web crawlers |\n",
+    "|depth:int | control crawling depth |\n",
+    "| downloads:int | number of downloads that are stored to the download folder. Since the crawler operations happen asynchronously, the process can result in any 10 of the visited URLs being retrieved (i.e. consecutive runs can result in different files being downloaded) |\n",
+    "| folder:str | folder where downloaded files are stored. If the folder is not empty, new files are  added or replace the existing ones with the same URLs |"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%capture\n",
+    "\n",
+    "## must enable nested asynchronous io in a notebook as the crawler uses \n",
+    "## coroutine to speed up acquisition and downloads\n",
+    "\n",
+    "import nest_asyncio\n",
+    "nest_asyncio.apply()\n",
+    "\n",
+    "from dpk_web2parquet.transform import Web2Parquet\n",
+    "\n",
+    "Web2Parquet(urls= [MY_CONFIG.CRAWL_URL_BASE],\n",
+    "                    depth=MY_CONFIG.CRAWL_MAX_DEPTH, \n",
+    "                    downloads=MY_CONFIG.CRAWL_MAX_DOWNLOADS,\n",
+    "                    folder=MY_CONFIG.INPUT_DIR).transform()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "✅ web crawl completed.  Downloaded 20 files into 'input' directory\n"
+     ]
+    }
+   ],
+   "source": [
+    "\n",
+    "# list some files\n",
+    "\n",
+    "print (f\"✅ web crawl completed.  Downloaded {len(os.listdir(MY_CONFIG.INPUT_DIR))} files into '{MY_CONFIG.INPUT_DIR}' directory\", )\n",
+    "\n",
+    "\n",
+    "# for item in os.listdir(MY_CONFIG.INPUT_DIR):\n",
+    "#     print(item)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "dpk-html-processing-2-022dev3-py312",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.7"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}