deploy: 4b42248

_sources/47_vision/22_vision_qwen.ipynb

@@ -37,60 +37,12 @@
    "metadata": {},
    "source": [
     "## Example image\n",
-    "First we load a medical tomography image."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "920cc018-fdd1-49c7-bda3-fdd269c0030b",
-   "metadata": {},
-   "source": [
-    "We will now send the image to ChatGPT and ask it the same questions."
+    "First we load a microscopy image. With such an example we can test if the model was trained on scientific microscopy data."
    ]
   },
   {
    "cell_type": "code",
    "execution_count": 2,
-   "id": "18422ecf-34f7-4ec3-a6b9-aa65098abab4",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "def prompt_qwen(prompt:str, image, model=\"Qwen/Qwen2-VL-7B-Instruct\"):\n",
-    "    \"\"\"A prompt helper function that sends a message to the server\n",
-    "    and returns only the text response.\n",
-    "    \"\"\"\n",
-    "    rgb_image = _img_to_rgb(image)\n",
-    "    byte_stream = numpy_to_bytestream(rgb_image)\n",
-    "    base64_image = base64.b64encode(byte_stream).decode('utf-8')\n",
-    "\n",
-    "    message = [{\"role\": \"user\", \"content\": [\n",
-    "        {\"type\": \"text\", \"text\": prompt},\n",
-    "        {\n",
-    "        \"type\": \"image_url\",\n",
-    "        \"image_url\": {\n",
-    "            \"url\": f\"data:image/png;base64,{base64_image}\"\n",
-    "        }\n",
-    "    }]}]\n",
-    "            \n",
-    "    # setup connection to the LLM\n",
-    "    client = openai.OpenAI(base_url=\"https://llm.scads.ai/v1\",\n",
-    "                           api_key=os.environ.get('SCADSAI_API_KEY'))\n",
-    "    \n",
-    "    # submit prompt\n",
-    "    response = client.chat.completions.create(\n",
-    "        model=model,\n",
-    "        messages=message\n",
-    "    )\n",
-    "    \n",
-    "    # extract answer\n",
-    "    return response.choices[0].message.content"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
    "id": "498ccb1b-e43c-4063-8962-3941141dda58",
    "metadata": {
     "tags": []
@@ -169,7 +121,7 @@
        "                   [11, 15,  5]]], dtype=uint8)"
       ]
      },
-     "execution_count": 3,
+     "execution_count": 2,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -179,6 +131,54 @@
     "stackview.insight(hela_cells)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "920cc018-fdd1-49c7-bda3-fdd269c0030b",
+   "metadata": {},
+   "source": [
+    "We will now send the image to the LLM server and ask it the some questions."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "18422ecf-34f7-4ec3-a6b9-aa65098abab4",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "def prompt_qwen(prompt:str, image, model=\"Qwen/Qwen2-VL-7B-Instruct\"):\n",
+    "    \"\"\"A prompt helper function that sends a message to the server\n",
+    "    and returns only the text response.\n",
+    "    \"\"\"\n",
+    "    rgb_image = _img_to_rgb(image)\n",
+    "    byte_stream = numpy_to_bytestream(rgb_image)\n",
+    "    base64_image = base64.b64encode(byte_stream).decode('utf-8')\n",
+    "\n",
+    "    message = [{\"role\": \"user\", \"content\": [\n",
+    "        {\"type\": \"text\", \"text\": prompt},\n",
+    "        {\n",
+    "        \"type\": \"image_url\",\n",
+    "        \"image_url\": {\n",
+    "            \"url\": f\"data:image/png;base64,{base64_image}\"\n",
+    "        }\n",
+    "    }]}]\n",
+    "            \n",
+    "    # setup connection to the LLM\n",
+    "    client = openai.OpenAI(base_url=\"https://llm.scads.ai/v1\",\n",
+    "                           api_key=os.environ.get('SCADSAI_API_KEY'))\n",
+    "    \n",
+    "    # submit prompt\n",
+    "    response = client.chat.completions.create(\n",
+    "        model=model,\n",
+    "        messages=message\n",
+    "    )\n",
+    "    \n",
+    "    # extract answer\n",
+    "    return response.choices[0].message.content"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": 4,
@@ -190,16 +190,11 @@
     {
      "data": {
       "text/markdown": [
-       "The image appears to be a microscopic view of biological cells, likely taken using fluorescence microscopy. Here are the key elements visible in the image:\n",
-       "\n",
-       "1. **Three Cells**: Three cells are prominently featured in the image.\n",
-       "2. **Nuclei**: Each cell has a large, blue, oval-shaped structure within it which seems to represent the nucleus.\n",
-       "3. **Cytoplasm**: Surrounding the nuclei is a lighter area that likely represents the cytoplasm.\n",
-       "4. **Stained Structures**: The cells have various structures stained with different colors:\n",
-       "   - Red or Fluorescent Dots**: These could be organelles such as vesicles or mitochondria within the cytoplasm of the cells.\n",
-       "   - Green Fluorescent Structures**: These could represent the cell membrane or other cellular components that have been stained differently for visualization.\n",
+       "The image appears to be a microscopic view of biological cells, possibly analyzed using fluorescence microscopy techniques. The cells are stained to highlight different structures:\n",
        "\n",
-       "This type of staining often aids researchers in understanding the distribution and movement of specific molecules within cells or tissue samples."
+       "1. **Blue Areas**: These represent the nuclei of the cells, typically stained with a dye like DAPI or Hoechst that binds to DNA.\n",
+       "2. **Green Structures**: These are likely the plasma membranes of the cells, as visualized by a fluorescence stain like FM1-43, which binds to lipid-bilayer membranes给人。\n",
+       "3. **Red Dots**: These are clusters or puncta that might represent specific signaling molecules or proteins, possibly related to cytokines or receptors participating in cell signaling pathways"
       ],
       "text/plain": [
        "<IPython.core.display.Markdown object>"