langchain/docs/docs/integrations/document_loaders/oracleai.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Oracle AI Vector Search: Document Processing\n",
    "Oracle AI Vector Search is designed for Artificial Intelligence (AI) workloads that allows you to query data based on semantics, rather than keywords. One of the biggest benefit of Oracle AI Vector Search is that semantic search on unstructured data can be combined with relational search on business data in one single system. This is not only powerful but also significantly more effective because you don't need to add a specialized vector database, eliminating the pain of data fragmentation between multiple systems.\n",
    "\n",
    "The guide demonstrates how to use Document Processing Capabilities within Oracle AI Vector Search to load and chunk documents using OracleDocLoader and OracleTextSplitter respectively."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prerequisites\n",
    "\n",
    "Please install Oracle Python Client driver to use Langchain with Oracle AI Vector Search. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# pip install oracledb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Connect to Oracle Database\n",
    "The following sample code will show how to connect to Oracle Database. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "import oracledb\n",
    "\n",
    "# please update with your username, password, hostname and service_name\n",
    "username = \"<username>\"\n",
    "password = \"<password>\"\n",
    "dsn = \"<hostname>/<service_name>\"\n",
    "\n",
    "try:\n",
    "    conn = oracledb.connect(user=username, password=password, dsn=dsn)\n",
    "    print(\"Connection successful!\")\n",
    "except Exception as e:\n",
    "    print(\"Connection failed!\")\n",
    "    sys.exit(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's create a table and insert some sample docs to test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    cursor = conn.cursor()\n",
    "\n",
    "    drop_table_sql = \"\"\"drop table if exists demo_tab\"\"\"\n",
    "    cursor.execute(drop_table_sql)\n",
    "\n",
    "    create_table_sql = \"\"\"create table demo_tab (id number, data clob)\"\"\"\n",
    "    cursor.execute(create_table_sql)\n",
    "\n",
    "    insert_row_sql = \"\"\"insert into demo_tab values (:1, :2)\"\"\"\n",
    "    rows_to_insert = [\n",
    "        (\n",
    "            1,\n",
    "            \"If the answer to any preceding questions is yes, then the database stops the search and allocates space from the specified tablespace; otherwise, space is allocated from the database default shared temporary tablespace.\",\n",
    "        ),\n",
    "        (\n",
    "            2,\n",
    "            \"A tablespace can be online (accessible) or offline (not accessible) whenever the database is open.\\nA tablespace is usually online so that its data is available to users. The SYSTEM tablespace and temporary tablespaces cannot be taken offline.\",\n",
    "        ),\n",
    "        (\n",
    "            3,\n",
    "            \"The database stores LOBs differently from other data types. Creating a LOB column implicitly creates a LOB segment and a LOB index. The tablespace containing the LOB segment and LOB index, which are always stored together, may be different from the tablespace containing the table.\\nSometimes the database can store small amounts of LOB data in the table itself rather than in a separate LOB segment.\",\n",
    "        ),\n",
    "    ]\n",
    "    cursor.executemany(insert_row_sql, rows_to_insert)\n",
    "\n",
    "    conn.commit()\n",
    "\n",
    "    print(\"Table created and populated.\")\n",
    "    cursor.close()\n",
    "except Exception as e:\n",
    "    print(\"Table creation failed.\")\n",
    "    cursor.close()\n",
    "    conn.close()\n",
    "    sys.exit(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load Documents\n",
    "The users can load the documents from Oracle Database or a file system or both. They just need to set the loader parameters accordingly. Please refer to the Oracle AI Vector Search Guide book for complete information about these parameters.\n",
    "\n",
    "The main benefit of using OracleDocLoader is that it can handle 150+ different file formats. You don't need to use different types of loader for different file formats. Here is the list of the formats that we support: [Oracle Text Supported Document Formats](https://docs.oracle.com/en/database/oracle/oracle-database/23/ccref/oracle-text-supported-document-formats.html)\n",
    "\n",
    "The following sample code will show how to do that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.document_loaders.oracleai import OracleDocLoader\n",
    "from langchain_core.documents import Document\n",
    "\n",
    "\"\"\"\n",
    "# loading a local file\n",
    "loader_params = {}\n",
    "loader_params[\"file\"] = \"<file>\"\n",
    "\n",
    "# loading from a local directory\n",
    "loader_params = {}\n",
    "loader_params[\"dir\"] = \"<directory>\"\n",
    "\"\"\"\n",
    "\n",
    "# loading from Oracle Database table\n",
    "loader_params = {\n",
    "    \"owner\": \"<owner>\",\n",
    "    \"tablename\": \"demo_tab\",\n",
    "    \"colname\": \"data\",\n",
    "}\n",
    "\n",
    "\"\"\" load the docs \"\"\"\n",
    "loader = OracleDocLoader(conn=conn, params=loader_params)\n",
    "docs = loader.load()\n",
    "\n",
    "\"\"\" verify \"\"\"\n",
    "print(f\"Number of docs loaded: {len(docs)}\")\n",
    "# print(f\"Document-0: {docs[0].page_content}\") # content"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Split Documents\n",
    "The documents can be in different sizes: small, medium, large, or very large. The users like to split/chunk their documents into smaller pieces to generate embeddings. There are lots of different splitting customizations the users can do. Please refer to the Oracle AI Vector Search Guide book for complete information about these parameters.\n",
    "\n",
    "The following sample code will show how to do that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.document_loaders.oracleai import OracleTextSplitter\n",
    "from langchain_core.documents import Document\n",
    "\n",
    "\"\"\"\n",
    "# Some examples\n",
    "# split by chars, max 500 chars\n",
    "splitter_params = {\"split\": \"chars\", \"max\": 500, \"normalize\": \"all\"}\n",
    "\n",
    "# split by words, max 100 words\n",
    "splitter_params = {\"split\": \"words\", \"max\": 100, \"normalize\": \"all\"}\n",
    "\n",
    "# split by sentence, max 20 sentences\n",
    "splitter_params = {\"split\": \"sentence\", \"max\": 20, \"normalize\": \"all\"}\n",
    "\"\"\"\n",
    "\n",
    "# split by default parameters\n",
    "splitter_params = {\"normalize\": \"all\"}\n",
    "\n",
    "# get the splitter instance\n",
    "splitter = OracleTextSplitter(conn=conn, params=splitter_params)\n",
    "\n",
    "list_chunks = []\n",
    "for doc in docs:\n",
    "    chunks = splitter.split_text(doc.page_content)\n",
    "    list_chunks.extend(chunks)\n",
    "\n",
    "\"\"\" verify \"\"\"\n",
    "print(f\"Number of Chunks: {len(list_chunks)}\")\n",
    "# print(f\"Chunk-0: {list_chunks[0]}\") # content"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### End to End Demo\n",
    "Please refer to our complete demo guide [Oracle AI Vector Search End-to-End Demo Guide](https://github.com/langchain-ai/langchain/tree/master/cookbook/oracleai_demo.ipynb) to build an end to end RAG pipeline with the help of Oracle AI Vector Search.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}