From 25252f254b6640380ed4d0908e17456a61b4c550 Mon Sep 17 00:00:00 2001 From: dajiaohuang Date: Tue, 6 Oct 2026 23:41:11 +0800 Subject: [PATCH 1/3] Fix docs navigation and examples --- docs/docs/advanced/best-practices.md | 16 ++--- docs/docs/advanced/visualize.md | 2 +- docs/docs/api/index.md | 1 - docs/docs/api/query.md | 2 +- docs/docs/concepts/.pages | 4 -- docs/docs/getting-started/quickstart.md | 6 +- .../managed-credentials/aws/cors.md | 2 +- .../managed-credentials/aws/provisioning.md | 22 +++---- .../managed-credentials/azure/cors.md | 4 +- .../managed-credentials/azure/provisioning.md | 28 ++++---- .../azure/workload-identities.md | 14 ++-- .../managed-credentials/gcp/provisioning.md | 22 +++---- .../guide/deep-learning/async-data-loader.md | 4 +- .../guide/deep-learning/deep-learning.ipynb | 64 +++---------------- docs/docs/guide/deep-learning/mmseg.md | 2 +- docs/docs/guide/rag.md | 42 +++--------- docs/docs/guide/v3-conversion.md | 5 +- docs/docs/llms.txt | 4 +- 18 files changed, 89 insertions(+), 155 deletions(-) delete mode 100644 docs/docs/concepts/.pages diff --git a/docs/docs/advanced/best-practices.md b/docs/docs/advanced/best-practices.md index 9395bb1642..26430f44a4 100644 --- a/docs/docs/advanced/best-practices.md +++ b/docs/docs/advanced/best-practices.md @@ -78,7 +78,7 @@ The second method is more efficient as the Deep Lake columnar format is handling ### Avoid decompressing images when adding them to the dataset -Deep Lake [Image](../../api/types/#deeplake.types.Image) type supports multiple image formats including JPEG, PNG, and TIFF. When adding images to the dataset, no need to decompress images into `numpy.array` or `PIL.Image` objects. Instead you can pass raw bytes and Deep Lake will automatically decompress the images when reading them. +Deep Lake [Image](../api/types/#deeplake.types.Image) type supports multiple image formats including JPEG, PNG, and TIFF. When adding images to the dataset, no need to decompress images into `numpy.array` or `PIL.Image` objects. Instead you can pass raw bytes and Deep Lake will automatically decompress the images when reading them. ```python image_ds.add_column("images", deeplake.types.Image("uint8", "jpeg")) @@ -88,9 +88,9 @@ image_ds.append({"images": f.read()}) This will be much faster and use less memory compared to decompressing the images before adding them to the dataset. -### Use [Link Image](../../api/types/#deeplake.types.Link) for cloud storage images +### Use [Link Image](../api/types/#deeplake.types.Link) for cloud storage images -If the images are in the cloud storage and you want to avoid data duplication you can use [Link Image](../../api/types/#deeplake.types.Link) type. In this case you need to pass the image url. +If the images are in the cloud storage and you want to avoid data duplication you can use [Link Image](../api/types/#deeplake.types.Link) type. In this case you need to pass the image url. ```python link_image_ds.add_column("images", deeplake.types.Link(deeplake.types.Image())) @@ -111,7 +111,7 @@ Small data updates like `ds['column1'][i] = value` can be done in Deep Lake data ### Prefer opening datasets in read-only mode. -When you are not planning to modify the dataset, prefer opening the dataset with [open_read_only](../../api/dataset/#deeplake.open_read_only) method. This will prevent accidental modifications to the dataset and will improve the performance of the data access. +When you are not planning to modify the dataset, prefer opening the dataset with [open_read_only](../api/dataset/#deeplake.open_read_only) method. This will prevent accidental modifications to the dataset and will improve the performance of the data access. ### Prefer batch access instead of row by row access @@ -139,7 +139,7 @@ for i in range(len(column)): ### Use `query` for complex data filtering and search -Deep Lake supports SQL-like queries to filter and search the data. If you need to filter the data based on multiple columns or complex conditions, prefer using the [deeplake.DatasetView.query](../../api/dataset#deeplake.DatasetView) or [deeplake.query](../../api/query) method, instead of doing manual iteration and filtering. +Deep Lake supports SQL-like queries to filter and search the data. If you need to filter the data based on multiple columns or complex conditions, prefer using the [deeplake.DatasetView.query](../api/dataset#deeplake.DatasetView) or [deeplake.query](../api/query) method, instead of doing manual iteration and filtering. ### Avoid accessing the data of the whole column @@ -147,7 +147,7 @@ The column data can be accessed directly by `ds["column_name"][:]`. For the larg ### Consider using async data access -Deep Lake supports async data access and query. If your workflow allows async processing and benefits from that, consider using async data access. Please refer to the [Async Data Loader](../../guide/deep-learning/async-data-loader) guide for the details. +Deep Lake supports async data access and query. If your workflow allows async processing and benefits from that, consider using async data access. Please refer to the [Async Data Loader](../guide/deep-learning/async-data-loader) guide for the details. ## Storage and Data Management @@ -169,7 +169,7 @@ Deep Lake supports multiple storage backends, which are differentiated by url sc In memory datasets are not persisted and are useful for temporary data storage and testing. They are automatically deleted when the process is terminated. -If you created in memory dataset and you want to persist it you can use [deeplake.copy](../../advanced/sync/#copying-datasets) method to copy the dataset to the local or cloud storage. +If you created in memory dataset and you want to persist it you can use [deeplake.copy](../advanced/sync/#copying-datasets) method to copy the dataset to the local or cloud storage. ### Avoid local storage for large datasets @@ -177,7 +177,7 @@ Local storage shows better latency compared to the cloud dataset. However it is If you plan to store large data (>=100GB) cloud storage will be more efficient and reliable. -If you have local dataset and you want to move it to the cloud you can use [deeplake.copy](../../advanced/sync/#copying-datasets) method to copy the dataset to the cloud storage. +If you have local dataset and you want to move it to the cloud you can use [deeplake.copy](../advanced/sync/#copying-datasets) method to copy the dataset to the cloud storage. ### Prefer accessing the cloud storage from the same region diff --git a/docs/docs/advanced/visualize.md b/docs/docs/advanced/visualize.md index 55d7773a20..d15881e0e5 100644 --- a/docs/docs/advanced/visualize.md +++ b/docs/docs/advanced/visualize.md @@ -5,7 +5,7 @@ description: "Learn how to visualize Deep Lake datasets using the Deep Lake web # Dataset Visualization -Deep Lake has a web interface for visualizing, versioning, and querying machine learning datasets. It utilizes the Deep Lake format under-the-hood, and it can be connected to datasets stored in all Deep Lake [storage locations](../../getting-started/storage-and-creds/). +Deep Lake has a web interface for visualizing, versioning, and querying machine learning datasets. It utilizes the Deep Lake format under-the-hood, and it can be connected to datasets stored in all Deep Lake [storage locations](../getting-started/storage-and-creds/). ![Visualizer](images/visualize.jpg) diff --git a/docs/docs/api/index.md b/docs/docs/api/index.md index 2a3c298efd..55acaa4590 100644 --- a/docs/docs/api/index.md +++ b/docs/docs/api/index.md @@ -16,7 +16,6 @@ This reference documents the Python API of Deep Lake. - [Query Language](query.md): Complete TQL syntax and operations - [Metadata](metadata.md): Dataset and column metadata management - [Version Control](version_control.md): Version control, history, branches, and tags -- [Miscellaneous](misc.md): Additional auxiliary functionality ## Index Management diff --git a/docs/docs/api/query.md b/docs/docs/api/query.md index 0f8cd46af4..162df87df1 100644 --- a/docs/docs/api/query.md +++ b/docs/docs/api/query.md @@ -229,7 +229,7 @@ results = deeplake.query(""" l.label FROM "s3://bucket1/images" AS i JOIN "gcs://bucket2/embeddings" AS e ON i.id = e.image_id - JOIN "azure://container/labels" AS l ON i.id = l.image_id + JOIN "azure:////labels" AS l ON i.id = l.image_id WHERE l.confidence > 0.9 ORDER BY COSINE_SIMILARITY(e.embedding, ARRAY[0.1, 0.2, 0.3]) DESC LIMIT 100 diff --git a/docs/docs/concepts/.pages b/docs/docs/concepts/.pages deleted file mode 100644 index c764a8509a..0000000000 --- a/docs/docs/concepts/.pages +++ /dev/null @@ -1,4 +0,0 @@ -nav: - - Architecture: architecture.md - - Data Model: data-model.md - - Performance: performance.md \ No newline at end of file diff --git a/docs/docs/getting-started/quickstart.md b/docs/docs/getting-started/quickstart.md index c50f79a377..940b2f4506 100644 --- a/docs/docs/getting-started/quickstart.md +++ b/docs/docs/getting-started/quickstart.md @@ -55,7 +55,7 @@ ds = deeplake.create("path/to/dataset") # Or create in cloud storage ds = deeplake.create("s3://my-bucket/dataset") ds = deeplake.create("gcs://my-bucket/dataset") -ds = deeplake.create("azure://container/dataset") +ds = deeplake.create("azure:////") ``` ## Adding Data @@ -179,8 +179,8 @@ cats = future.result() ## Next Steps -- Explore [RAG applications](../../guide/rag) -- Check out [Deep Learning integration](../../guide/deep-learning/deep-learning) +- Explore [RAG applications](../guide/rag.md) +- Check out [Deep Learning integration](../guide/deep-learning/deep-learning.ipynb) ## Support diff --git a/docs/docs/getting-started/storage-and-creds/managed-credentials/aws/cors.md b/docs/docs/getting-started/storage-and-creds/managed-credentials/aws/cors.md index 0d8f1640c9..2d1459f678 100644 --- a/docs/docs/getting-started/storage-and-creds/managed-credentials/aws/cors.md +++ b/docs/docs/getting-started/storage-and-creds/managed-credentials/aws/cors.md @@ -18,7 +18,7 @@ In order to visualize Deep Lake datasets stored in your own S3 buckets in the [D "HEAD" ], "AllowedOrigins": [ - "*.activeloop.ai" + "https://*.activeloop.ai" ], "ExposeHeaders": [] } diff --git a/docs/docs/getting-started/storage-and-creds/managed-credentials/aws/provisioning.md b/docs/docs/getting-started/storage-and-creds/managed-credentials/aws/provisioning.md index b0fd251e08..dab9980906 100644 --- a/docs/docs/getting-started/storage-and-creds/managed-credentials/aws/provisioning.md +++ b/docs/docs/getting-started/storage-and-creds/managed-credentials/aws/provisioning.md @@ -17,11 +17,11 @@ The most secure method for connecting data from your AWS account to Deep Lake is 3\. In the left nav, open the `Policies` under `Access management` and on `Create policy` on the right. -
+
5\. Select the `JSON` tab instead of `Visual editor`. -
+
6\. Replace the code in the editor with the code below. Replace `BUCKET_NAME` with the bucket names for which you want to grant role-based access: @@ -50,17 +50,17 @@ The most secure method for connecting data from your AWS account to Deep Lake is 7\. On the bottom right, click `Next: Tags` (create tags if needed) and `Next: Preview`, enter the policy `name` and `description`, and click `Create policy` -
+
#### Step 2: Create the AWS IAM Role 1\. On the `IAM` page, in the left nav, open the `Roles` under `Access management`, and click `Create role` on the right. -
+
3\. Select `Custom trust policy` from the list of options. -
+
4\. Replace the policy definition with the code below and click `Next` @@ -83,11 +83,11 @@ The most secure method for connecting data from your AWS account to Deep Lake is 5\. From the provided policy list, select the previously created policy from Step 1 and click `Next` -
+
6\. Set the `name` and `description` for the role and click `Create role` at the bottom. -
+
#### Step 3: Grant Access to AWS KMS Key (**only for buckets that are encrypted with customer managed KMS keys**) @@ -95,11 +95,11 @@ The most secure method for connecting data from your AWS account to Deep Lake is 2\. Open the bucket Properties -
+
3\. Scroll down to Default encryption and copy the `AWS KMS key ARN` -
+
4\. In the Policy creation step (Step 1, Sub-step 6), use the JSON below in the policy statement, and replace `YOUR_KMS_KEY_ARN` with the copied Key ARN for the encrypted bucket. @@ -129,7 +129,7 @@ The most secure method for connecting data from your AWS account to Deep Lake is "kms:DescribeKey" ], "Resource": [ - "YOUR_KMS_KEY_ARN” + "YOUR_KMS_KEY_ARN" ] } ] @@ -143,4 +143,4 @@ See the first video in the [managed credentials overview](../index.md) ## Next Steps -- [Enabling CORS in AWS S3](cors.md) \ No newline at end of file +- [Enabling CORS in AWS S3](cors.md) diff --git a/docs/docs/getting-started/storage-and-creds/managed-credentials/azure/cors.md b/docs/docs/getting-started/storage-and-creds/managed-credentials/azure/cors.md index cc67bcf7e5..37a523b6d6 100644 --- a/docs/docs/getting-started/storage-and-creds/managed-credentials/azure/cors.md +++ b/docs/docs/getting-started/storage-and-creds/managed-credentials/azure/cors.md @@ -19,7 +19,7 @@ Cross-Origin Resource Sharing (CORS) is typically enabled by default in Azure. I 4\. Add the following items to the permissions. -
+
| Allowed origins | Allowed methods | Allowed headers | | ------------------------- | --------------- | --------------- | @@ -27,4 +27,4 @@ Cross-Origin Resource Sharing (CORS) is typically enabled by default in Azure. I ## Next Steps -- [Provisioning Federated Credentials](provisioning.md) \ No newline at end of file +- [Provisioning Federated Credentials](provisioning.md) diff --git a/docs/docs/getting-started/storage-and-creds/managed-credentials/azure/provisioning.md b/docs/docs/getting-started/storage-and-creds/managed-credentials/azure/provisioning.md index 86d2cf64c9..838fd7fddc 100644 --- a/docs/docs/getting-started/storage-and-creds/managed-credentials/azure/provisioning.md +++ b/docs/docs/getting-started/storage-and-creds/managed-credentials/azure/provisioning.md @@ -17,23 +17,23 @@ The most secure method for connecting data from your Azure storage to Deep Lake 3\. Click on `Register an application` or `New registration`. -
+
4\. Enter the `Name` and `Supported account type` (all are supported in Deep Lake) and click `Register` -
+
5\. In the application console, click `Certificates & secrets`. -
+
6\. Click on `Federated credentials` and `Add credential`. -
+
7\. Click on `Select scenario` and select `Other issuer`. -
+
8\. Enter the following information in the form, and click `Add`. @@ -44,7 +44,7 @@ The most secure method for connecting data from your Azure storage to Deep Lake * **Description (optional):** enter description a with your choice * **Audience:** us-east-1:57e5de2f-e2ec-4514-b9b0-f3bb8c4283c3 -
+
#### Step 2a: Apply the Application Credentials to your Azure storage account @@ -54,23 +54,23 @@ The most secure method for connecting data from your Azure storage to Deep Lake 2\. Select the `Storage account` to which you want to add Application Credentials. -
+
3\. Select `Access Control (IAM)` and click `Add`, and `select Add role assignment`. -
+
4\. Search and select `Storage Blob Data Contributor` under the role names and click `Next`. -
+
5\. Click on the `Select members` link, and in the tab that opens up on the right, search by name and select the application you created in Step 1. Click `Select` at the bottom of the page. -
+
6\. The application should appear in the list of Members, at which point you can click `Review + assign`. -
+
#### Step 2b: Apply the Application Credentials to a specific Azure contained in your Azure storage account @@ -78,15 +78,15 @@ The most secure method for connecting data from your Azure storage to Deep Lake 2\. Select the `Storage account` to which you want to add Application Credentials. -
+
3\. Select the `Container` to which you add the Application Credentials. -
+
4\. Select `Access Control (IAM)` and click `Add`, and `select Add role assignment`. -
+
#### IMPORTANT TO PERFORM STEPS BELOW TO COMPLETE 2b - PLEASE DO NOT SKIP diff --git a/docs/docs/getting-started/storage-and-creds/managed-credentials/azure/workload-identities.md b/docs/docs/getting-started/storage-and-creds/managed-credentials/azure/workload-identities.md index 2bb86c2006..a87cb2d49b 100644 --- a/docs/docs/getting-started/storage-and-creds/managed-credentials/azure/workload-identities.md +++ b/docs/docs/getting-started/storage-and-creds/managed-credentials/azure/workload-identities.md @@ -23,15 +23,15 @@ Set up a Workload Identity using the following steps: 1. Navigate to Managed Identities in Azure - + 1. Click `Create` a Managed Identity - + 1. Select the `Subscription` and `Resource Group` containing the workload, and give the Managed Identity a `Name`. Click `Review + Create`. - + ## Step 2: Attached the Azure Managed Identity to your workload @@ -40,17 +40,17 @@ When creating or updating a resource that will serve as the Client running Deep For example, in Azure Machine Learning Studio, when creating a compute instance, toggle `Assign Identity` and select the `Managed Identity` from Step 1. - + ## Step 3: Create a Deep Lake Workload Identity using the Azure Managed Identity Navigate to the `Permissions` tab for your organization in the [Deep Lake App](https://app.activeloop.ai/), locate the `Workload Identities`, and select `Add`. - + Specify a `Display Name`, `Client ID` (for the Managed Identity), and `Tenant ID`. The `Client ID` can be found in the main page for the Managed Identity, and the `Tenant ID` can be found in `Tenant Properties` in Azure. Click `Add`. - + ## Step 4: Run the workload @@ -76,4 +76,4 @@ Specifying the `AZURE_CLIENT_ID` is not necessary in some environments because t For a compute instance in the Azure Machine Learning Studio, the Client ID can be found in instance settings below: - + diff --git a/docs/docs/getting-started/storage-and-creds/managed-credentials/gcp/provisioning.md b/docs/docs/getting-started/storage-and-creds/managed-credentials/gcp/provisioning.md index ca96b7ed8f..fd81f476aa 100644 --- a/docs/docs/getting-started/storage-and-creds/managed-credentials/gcp/provisioning.md +++ b/docs/docs/getting-started/storage-and-creds/managed-credentials/gcp/provisioning.md @@ -7,7 +7,7 @@ description: "Provision Google Cloud Resources For Use With Deep Lake, Enabling ## Setting up Federated Credentials in Google Cloud Platform -The most secure method for connecting data from your Azure storage to Deep Lake is using Federated Credentials, which are set up using the steps below: +The most secure method for connecting data from your Google Cloud Storage bucket to Deep Lake is using Federated Credentials, which are set up using the steps below: #### Step 1: Create Google Cloud Service Account @@ -15,37 +15,37 @@ The most secure method for connecting data from your Azure storage to Deep Lake 2\. Navigate to `IAM & Admin` -> `Service Accounts` -> `CREATE SERVICE ACCOUNT` -
-
+
+
3\. Enter the `service account id`, and optional `name` and `description`. Make sure to copy the email address and and click on `CREATE AND CONTINUE`. -
+
4\. Click `CONTINUE` without entering any information. -
+
5\. Enter `activeloop-platform@activeloop-saas-iam.iam.gserviceaccount.com` in the `Service account users role` and click `DONE`. -
+
#### Step 2: Grant Access to the bucket using a Service Account Principal 1\. Navigate to `Cloud Storage` and `Buckets`. -
+
2\. Select `Edit Access` for the bucket you want to connect to Activeloop. -
+
3\. Select `Add Principal`. -
+
4\. Enter the `Service Account Email`, select the role as `Storage Object Admin`, and click `Save`. If the bucket is encrypted with customer managed KMS key, then `Cloud KMS CryptoKey Encrypter/Decrypter` should be added in the `Role` field as well. -
+
#### Step 3: Enter the Service Account Email (Step 2) into the Activeloop App @@ -53,4 +53,4 @@ See the first video in the [managed credentials overview](../index.md) ## Next Steps -- [Enabling CORS in GCP](cors.md) \ No newline at end of file +- [Enabling CORS in GCP](cors.md) diff --git a/docs/docs/guide/deep-learning/async-data-loader.md b/docs/docs/guide/deep-learning/async-data-loader.md index 0d109c79b1..2866ed041b 100644 --- a/docs/docs/guide/deep-learning/async-data-loader.md +++ b/docs/docs/guide/deep-learning/async-data-loader.md @@ -66,11 +66,13 @@ lock = Lock() index = -1 def generate_data(ds: deeplake.Dataset): total_count = len(ds) + if total_count == 0: + return global index while True: idx = 0 with lock: - index = (index + 1) % (total_count - 1) + index = (index + 1) % total_count idx = index yield ds[idx] diff --git a/docs/docs/guide/deep-learning/deep-learning.ipynb b/docs/docs/guide/deep-learning/deep-learning.ipynb index 332ccca3c3..caec3a728f 100644 --- a/docs/docs/guide/deep-learning/deep-learning.ipynb +++ b/docs/docs/guide/deep-learning/deep-learning.ipynb @@ -77,27 +77,9 @@ }, { "cell_type": "code", - "execution_count": 4, + "execution_count": null, "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Dataset(columns=(images,labels,boxes), length=6471)\n", - "\u001B[1m+------+--------------------------------------------+\n", - "|column| type |\n", - "+------+--------------------------------------------+\u001B[0m\n", - "|images| array(dtype=uint8, shape=[None,None,None]) |\n", - "+------+--------------------------------------------+\n", - "|labels| array(dtype=uint32, shape=[None]) |\n", - "+------+--------------------------------------------+\n", - "|boxes |array(dtype=float32, shape=[None,None,None])|\n", - "+------+--------------------------------------------+\n", - "\n" - ] - } - ], + "outputs": [], "source": [ "ds.summary()" ] @@ -159,24 +141,14 @@ }, { "cell_type": "code", - "execution_count": 57, + "execution_count": null, "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - " % Total % Received % Xferd Average Speed Time Time Time Current\n", - " Dload Upload Total Spent Left Speed\n", - "100 167k 0 167k 0 0 426k 0 --:--:-- --:--:-- --:--:-- 425k\n" - ] - } - ], + "outputs": [], "source": [ "# Download dataset\n", "from IPython.display import clear_output\n", - "# !wget https://github.com/activeloopai/examples/blob/main/colabs/starting_data/animals.tar\n", - "!curl -L -o animals.tar https://github.com/activeloopai/examples/blob/main/colabs/starting_data/animals.tar" + "# !wget https://raw.githubusercontent.com/activeloopai/examples/main/colabs/starting_data/animals.tar\n", + "!curl -L -o animals.tar https://raw.githubusercontent.com/activeloopai/examples/main/colabs/starting_data/animals.tar" ] }, { @@ -241,12 +213,12 @@ }, { "cell_type": "code", - "execution_count": 59, + "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Find the class_names and list of files that need to be uploaded\n", - "dataset_folder = '/Users/istranic/ActiveloopCode/Datasets/animals'\n", + "dataset_folder = \"animals\"\n", "\n", "# Find the subfolders, but filter additional files like DS_Store that are added on Mac machines.\n", "class_names = [item for item in os.listdir(dataset_folder) if os.path.isdir(os.path.join(dataset_folder, item))]\n", @@ -284,25 +256,9 @@ }, { "cell_type": "code", - "execution_count": 68, + "execution_count": null, "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Dataset(columns=(images,labels), length=0)\n", - "\u001B[1m+------+------------------------------------------+\n", - "|column| type |\n", - "+------+------------------------------------------+\u001B[0m\n", - "|images|array(dtype=uint8, shape=[None,None,None])|\n", - "+------+------------------------------------------+\n", - "|labels| array(dtype=uint32, shape=[None]) |\n", - "+------+------------------------------------------+\n", - "\n" - ] - } - ], + "outputs": [], "source": [ "ds.summary()" ] diff --git a/docs/docs/guide/deep-learning/mmseg.md b/docs/docs/guide/deep-learning/mmseg.md index a8452ba963..74c3257dc5 100644 --- a/docs/docs/guide/deep-learning/mmseg.md +++ b/docs/docs/guide/deep-learning/mmseg.md @@ -111,7 +111,7 @@ data = dict( ), val=dict( pipeline=test_pipeline, - deeplake_path="hub://activeloop/coco-val-seg-mask"", + deeplake_path="hub://activeloop/coco-val-seg-mask", deeplake_tensors = {"img": "images", "gt_semantic_seg": "seg_masks"}, deeplake_dataloader={"shuffle": False, "batch_size": 1, "num_workers": 0, "drop_last": True} ) diff --git a/docs/docs/guide/rag.md b/docs/docs/guide/rag.md index 7106ef92d6..092bfdd6a6 100644 --- a/docs/docs/guide/rag.md +++ b/docs/docs/guide/rag.md @@ -580,7 +580,7 @@ class Document(BaseModel): data: Dict[str, Any] score: Optional[float] = None -def softmax(retrieved_score: list[float], max_weight: int = 700) -> Dict[str, Document]: +def softmax(retrieved_score: list[float], max_weight: int = 700) -> list[float]: # Compute the exponentials exp_scores = [math.exp(min(score, max_weight)) for score in retrieved_score] @@ -672,7 +672,7 @@ def fusion(docs_vs: Dict[str, Document], docs_bm25: Dict[str, Document]) -> Dict VECTOR_WEIGHT = 0.5 LEXICAL_WEIGHT = 0.5 - results: Dict[str, Dict[str, Document]] = {} + results: Dict[str, Document] = {} for k in set(docs_vs) | set(docs_bm25): @@ -689,19 +689,9 @@ results = fusion(docs_vs, docs_bm25) print(results) ``` -Output: -``` -{'2637': Document(id='2637', data={'restaurant_name': 'Mifen101 花溪米粉王', 'restaurant_review': 'Feel like I’m back in China.'}, score=0.013747293509625419), - '5136': Document(id='5136', data={'restaurant_name': 'Scratch', 'restaurant_review': 'Just had drinks. They were good!'}, score=0.024505473374994282), - '17426': Document(id='17426', data={'restaurant_name': "St. Stephen's Green", 'restaurant_review': 'Good drinks an easy going bartenders'}, score=0.024579178342433523), - '17444': Document(id='17444', data={'restaurant_name': "St. Stephen's Green", 'restaurant_review': 'Good drinks, good food'}, score=0.02475096426920331), - '2496': Document(id='2496', data={'restaurant_name': 'Seasons Noodles & Dumplings Garden', 'restaurant_review': 'Comfort food, excellent service! Feel like back to home.'}, score=0.005430922978171003), - '4022': Document(id='4022', data={'restaurant_name': 'Eureka! Mountain View', 'restaurant_review': 'Good drinks and burgers'}, score=0.0246334319067476), - '3518': Document(id='3518', data={'restaurant_name': 'Olympus Caffe & Bakery', 'restaurant_review': 'I like the garden to sit down with friends and have a drink.'}, score=0.08915287739833623), - '17502': Document(id='17502', data={'restaurant_name': "St. Stephen's Green", 'restaurant_review': 'Nice place for a drink'}, score=0.02653095210662131), - '11383': Document(id='11383', data={'restaurant_name': 'Ludwigs Biergarten Mountain View', 'restaurant_review': 'Beer is fresh tables are big feel like a proper beer garden'}, score=0.011447537567869991), - '10788': Document(id='10788', data={'restaurant_name': 'Casa Lupe', 'restaurant_review': 'Run by a family that makes you feel like part of the family. Awesome food. I love their wet Chili Verde burritos'}, score=0.00522136854599736)} -``` +The exact scores depend on the retrieved rows; use the sorted results below. + + We sort the results dictionary by each document's combined score in descending order, ensuring that the highest-ranking documents appear first. @@ -710,19 +700,7 @@ sorted_documents = dict(sorted(results.items(), key=lambda item: item[1].score, print(sorted_documents) ``` -Output: -``` -{'3518': Document(id='3518', data={'restaurant_name': 'Olympus Caffe & Bakery', 'restaurant_review': 'I like the garden to sit down with friends and have a drink.'}, score=0.3566115095933449), - '17502': Document(id='17502', data={'restaurant_name': "St. Stephen's Green", 'restaurant_review': 'Nice place for a drink'}, score=0.10612380842648524), - '17444': Document(id='17444', data={'restaurant_name': "St. Stephen's Green", 'restaurant_review': 'Good drinks, good food'}, score=0.09900385707681324), - '4022': Document(id='4022', data={'restaurant_name': 'Eureka! Mountain View', 'restaurant_review': 'Good drinks and burgers'}, score=0.0985337276269904), - '17426': Document(id='17426', data={'restaurant_name': "St. Stephen's Green", 'restaurant_review': 'Good drinks an easy going bartenders'}, score=0.09831671336973409), - '5136': Document(id='5136', data={'restaurant_name': 'Scratch', 'restaurant_review': 'Just had drinks. They were good!'}, score=0.09802189349997713), - '2637': Document(id='2637', data={'restaurant_name': 'Mifen101 花溪米粉王', 'restaurant_review': 'Feel like I’m back in China.'}, score=0.054989174038501676), - '11383': Document(id='11383', data={'restaurant_name': 'Ludwigs Biergarten Mountain View', 'restaurant_review': 'Beer is fresh tables are big feel like a proper beer garden'}, score=0.045790150271479965), - '2496': Document(id='2496', data={'restaurant_name': 'Seasons Noodles & Dumplings Garden', 'restaurant_review': 'Comfort food, excellent service! Feel like back to home.'}, score=0.02172369191268401), - '10788': Document(id='10788', data={'restaurant_name': 'Casa Lupe', 'restaurant_review': 'Run by a family that makes you feel like part of the family. Awesome food. I love their wet Chili Verde burritos'}, score=0.02088547418398944)} -``` +The sorted output contains the retrieved documents in descending fused score order. ### Show the results @@ -856,6 +834,7 @@ for el, score in zip(bm25_results, bm25s): ```python results = fusion(docs_vs, docs_bm25) +sorted_documents = dict(sorted(results.items(), key=lambda item: item[1].score, reverse=True)) ``` ```python @@ -863,8 +842,8 @@ for v in sorted_documents.values(): print(f"Restaurant name: {v.data['restaurant_name']} \nReview: {v.data['restaurant_review']}") ``` -Output: -``` +The printed rows depend on the contents of both datasets. + Restaurant name: Olympus Caffe & Bakery Review: I like the garden to sit down with friends and have a drink. Restaurant name: St. Stephen's Green @@ -1287,8 +1266,7 @@ Dataset(columns=(text,embedding), length=19719) ``` ```python -medical_dataset.add_column(name="embedding", dtype=types.Array(types.Float32(),dimensions=2)) -medical_dataset.commit() +# The copied dataset already has an `embedding` column, so reuse it. ``` 3. **Text Extraction** : diff --git a/docs/docs/guide/v3-conversion.md b/docs/docs/guide/v3-conversion.md index eb92c10aa0..d96429717f 100644 --- a/docs/docs/guide/v3-conversion.md +++ b/docs/docs/guide/v3-conversion.md @@ -105,6 +105,8 @@ deeplake.create = create ```python # 1. Create v4 dataset with desired schema +old_ds_path = "hub://org_name/v3_dataset" # Replace with the v3 dataset path to migrate. +batch_size = 32 ds = deeplake.create("s3://new/dataset") ds.add_column("embeddings", deeplake.types.Embedding(768)) ds.add_column("images", deeplake.types.Image()) @@ -117,7 +119,8 @@ source = deeplake.query(f'SELECT * FROM "{old_ds_path}"') for i in range(0, len(source), batch_size): batch = source[i:i+batch_size] ds.append(batch) - if i % 10000 == 0: + processed = i + len(batch) + if processed // 10000 > i // 10000 or processed == len(source): ds.commit() ``` diff --git a/docs/docs/llms.txt b/docs/docs/llms.txt index 885ff5e876..1fe530a5ec 100644 --- a/docs/docs/llms.txt +++ b/docs/docs/llms.txt @@ -122,7 +122,7 @@ SELECT AVG(features, axis=0) FROM dataset SELECT i.image, e.embedding, m.metadata FROM "s3://bucket1/images" AS i JOIN "gcs://bucket2/embeddings" AS e ON i.id = e.image_id -JOIN "azure://container/meta" AS m ON i.id = m.image_id +JOIN "azure:////meta" AS m ON i.id = m.image_id WHERE m.verified = true ORDER BY COSINE_SIMILARITY(e.embedding, ARRAY[0.1,0.2,0.3]) DESC ``` @@ -194,4 +194,4 @@ cats = view.query("SELECT * WHERE species = 'cat'") - [Column API](https://docs.deeplake.ai/api/column/): Column management and indexing - [Query API](https://docs.deeplake.ai/api/query/): TQL syntax and examples - [Data Types](https://docs.deeplake.ai/api/types/): All supported data types -- [TQL Reference](https://docs.deeplake.ai/advanced/tql/): Complete TQL syntax guide \ No newline at end of file +- [TQL Reference](https://docs.deeplake.ai/advanced/tql/): Complete TQL syntax guide From 1bc48214c2bf4a9e0a35626ab099a10009206e96 Mon Sep 17 00:00:00 2001 From: dajiaohuang Date: Wed, 7 Oct 2026 00:05:52 +0800 Subject: [PATCH 2/3] Clarify documentation examples --- docs/docs/guide/rag.md | 2 +- docs/docs/guide/v3-conversion.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/docs/guide/rag.md b/docs/docs/guide/rag.md index 092bfdd6a6..a2b935c8bd 100644 --- a/docs/docs/guide/rag.md +++ b/docs/docs/guide/rag.md @@ -842,7 +842,7 @@ for v in sorted_documents.values(): print(f"Restaurant name: {v.data['restaurant_name']} \nReview: {v.data['restaurant_review']}") ``` -The printed rows depend on the contents of both datasets. +Example output from one run (your rows may differ if either dataset changes): Restaurant name: Olympus Caffe & Bakery Review: I like the garden to sit down with friends and have a drink. diff --git a/docs/docs/guide/v3-conversion.md b/docs/docs/guide/v3-conversion.md index d96429717f..7f5380afe4 100644 --- a/docs/docs/guide/v3-conversion.md +++ b/docs/docs/guide/v3-conversion.md @@ -117,7 +117,7 @@ source = deeplake.query(f'SELECT * FROM "{old_ds_path}"') # 3. Migrate in batches with progress tracking for i in range(0, len(source), batch_size): - batch = source[i:i+batch_size] + batch = [row.to_dict() for row in source[i:i+batch_size]] ds.append(batch) processed = i + len(batch) if processed // 10000 > i // 10000 or processed == len(source): From f082b684978a7ced69bc16ca4fbd45350c8db2f7 Mon Sep 17 00:00:00 2001 From: dajiaohuang Date: Fri, 9 Oct 2026 00:06:19 +0800 Subject: [PATCH 3/3] Limit documentation errata to verified errors --- docs/docs/advanced/best-practices.md | 16 ++--- docs/docs/advanced/visualize.md | 2 +- .../managed-credentials/aws/provisioning.md | 20 +++---- .../managed-credentials/azure/cors.md | 6 +- .../managed-credentials/azure/provisioning.md | 28 ++++----- .../azure/workload-identities.md | 14 ++--- .../managed-credentials/gcp/provisioning.md | 20 +++---- .../guide/deep-learning/deep-learning.ipynb | 60 ++++++++++++++++--- docs/docs/guide/rag.md | 36 ++++++++--- docs/docs/llms.txt | 2 +- 10 files changed, 135 insertions(+), 69 deletions(-) diff --git a/docs/docs/advanced/best-practices.md b/docs/docs/advanced/best-practices.md index 26430f44a4..9395bb1642 100644 --- a/docs/docs/advanced/best-practices.md +++ b/docs/docs/advanced/best-practices.md @@ -78,7 +78,7 @@ The second method is more efficient as the Deep Lake columnar format is handling ### Avoid decompressing images when adding them to the dataset -Deep Lake [Image](../api/types/#deeplake.types.Image) type supports multiple image formats including JPEG, PNG, and TIFF. When adding images to the dataset, no need to decompress images into `numpy.array` or `PIL.Image` objects. Instead you can pass raw bytes and Deep Lake will automatically decompress the images when reading them. +Deep Lake [Image](../../api/types/#deeplake.types.Image) type supports multiple image formats including JPEG, PNG, and TIFF. When adding images to the dataset, no need to decompress images into `numpy.array` or `PIL.Image` objects. Instead you can pass raw bytes and Deep Lake will automatically decompress the images when reading them. ```python image_ds.add_column("images", deeplake.types.Image("uint8", "jpeg")) @@ -88,9 +88,9 @@ image_ds.append({"images": f.read()}) This will be much faster and use less memory compared to decompressing the images before adding them to the dataset. -### Use [Link Image](../api/types/#deeplake.types.Link) for cloud storage images +### Use [Link Image](../../api/types/#deeplake.types.Link) for cloud storage images -If the images are in the cloud storage and you want to avoid data duplication you can use [Link Image](../api/types/#deeplake.types.Link) type. In this case you need to pass the image url. +If the images are in the cloud storage and you want to avoid data duplication you can use [Link Image](../../api/types/#deeplake.types.Link) type. In this case you need to pass the image url. ```python link_image_ds.add_column("images", deeplake.types.Link(deeplake.types.Image())) @@ -111,7 +111,7 @@ Small data updates like `ds['column1'][i] = value` can be done in Deep Lake data ### Prefer opening datasets in read-only mode. -When you are not planning to modify the dataset, prefer opening the dataset with [open_read_only](../api/dataset/#deeplake.open_read_only) method. This will prevent accidental modifications to the dataset and will improve the performance of the data access. +When you are not planning to modify the dataset, prefer opening the dataset with [open_read_only](../../api/dataset/#deeplake.open_read_only) method. This will prevent accidental modifications to the dataset and will improve the performance of the data access. ### Prefer batch access instead of row by row access @@ -139,7 +139,7 @@ for i in range(len(column)): ### Use `query` for complex data filtering and search -Deep Lake supports SQL-like queries to filter and search the data. If you need to filter the data based on multiple columns or complex conditions, prefer using the [deeplake.DatasetView.query](../api/dataset#deeplake.DatasetView) or [deeplake.query](../api/query) method, instead of doing manual iteration and filtering. +Deep Lake supports SQL-like queries to filter and search the data. If you need to filter the data based on multiple columns or complex conditions, prefer using the [deeplake.DatasetView.query](../../api/dataset#deeplake.DatasetView) or [deeplake.query](../../api/query) method, instead of doing manual iteration and filtering. ### Avoid accessing the data of the whole column @@ -147,7 +147,7 @@ The column data can be accessed directly by `ds["column_name"][:]`. For the larg ### Consider using async data access -Deep Lake supports async data access and query. If your workflow allows async processing and benefits from that, consider using async data access. Please refer to the [Async Data Loader](../guide/deep-learning/async-data-loader) guide for the details. +Deep Lake supports async data access and query. If your workflow allows async processing and benefits from that, consider using async data access. Please refer to the [Async Data Loader](../../guide/deep-learning/async-data-loader) guide for the details. ## Storage and Data Management @@ -169,7 +169,7 @@ Deep Lake supports multiple storage backends, which are differentiated by url sc In memory datasets are not persisted and are useful for temporary data storage and testing. They are automatically deleted when the process is terminated. -If you created in memory dataset and you want to persist it you can use [deeplake.copy](../advanced/sync/#copying-datasets) method to copy the dataset to the local or cloud storage. +If you created in memory dataset and you want to persist it you can use [deeplake.copy](../../advanced/sync/#copying-datasets) method to copy the dataset to the local or cloud storage. ### Avoid local storage for large datasets @@ -177,7 +177,7 @@ Local storage shows better latency compared to the cloud dataset. However it is If you plan to store large data (>=100GB) cloud storage will be more efficient and reliable. -If you have local dataset and you want to move it to the cloud you can use [deeplake.copy](../advanced/sync/#copying-datasets) method to copy the dataset to the cloud storage. +If you have local dataset and you want to move it to the cloud you can use [deeplake.copy](../../advanced/sync/#copying-datasets) method to copy the dataset to the cloud storage. ### Prefer accessing the cloud storage from the same region diff --git a/docs/docs/advanced/visualize.md b/docs/docs/advanced/visualize.md index d15881e0e5..55d7773a20 100644 --- a/docs/docs/advanced/visualize.md +++ b/docs/docs/advanced/visualize.md @@ -5,7 +5,7 @@ description: "Learn how to visualize Deep Lake datasets using the Deep Lake web # Dataset Visualization -Deep Lake has a web interface for visualizing, versioning, and querying machine learning datasets. It utilizes the Deep Lake format under-the-hood, and it can be connected to datasets stored in all Deep Lake [storage locations](../getting-started/storage-and-creds/). +Deep Lake has a web interface for visualizing, versioning, and querying machine learning datasets. It utilizes the Deep Lake format under-the-hood, and it can be connected to datasets stored in all Deep Lake [storage locations](../../getting-started/storage-and-creds/). ![Visualizer](images/visualize.jpg) diff --git a/docs/docs/getting-started/storage-and-creds/managed-credentials/aws/provisioning.md b/docs/docs/getting-started/storage-and-creds/managed-credentials/aws/provisioning.md index dab9980906..33f5e84c27 100644 --- a/docs/docs/getting-started/storage-and-creds/managed-credentials/aws/provisioning.md +++ b/docs/docs/getting-started/storage-and-creds/managed-credentials/aws/provisioning.md @@ -17,11 +17,11 @@ The most secure method for connecting data from your AWS account to Deep Lake is 3\. In the left nav, open the `Policies` under `Access management` and on `Create policy` on the right. -
+
5\. Select the `JSON` tab instead of `Visual editor`. -
+
6\. Replace the code in the editor with the code below. Replace `BUCKET_NAME` with the bucket names for which you want to grant role-based access: @@ -50,17 +50,17 @@ The most secure method for connecting data from your AWS account to Deep Lake is 7\. On the bottom right, click `Next: Tags` (create tags if needed) and `Next: Preview`, enter the policy `name` and `description`, and click `Create policy` -
+
#### Step 2: Create the AWS IAM Role 1\. On the `IAM` page, in the left nav, open the `Roles` under `Access management`, and click `Create role` on the right. -
+
3\. Select `Custom trust policy` from the list of options. -
+
4\. Replace the policy definition with the code below and click `Next` @@ -83,11 +83,11 @@ The most secure method for connecting data from your AWS account to Deep Lake is 5\. From the provided policy list, select the previously created policy from Step 1 and click `Next` -
+
6\. Set the `name` and `description` for the role and click `Create role` at the bottom. -
+
#### Step 3: Grant Access to AWS KMS Key (**only for buckets that are encrypted with customer managed KMS keys**) @@ -95,11 +95,11 @@ The most secure method for connecting data from your AWS account to Deep Lake is 2\. Open the bucket Properties -
+
3\. Scroll down to Default encryption and copy the `AWS KMS key ARN` -
+
4\. In the Policy creation step (Step 1, Sub-step 6), use the JSON below in the policy statement, and replace `YOUR_KMS_KEY_ARN` with the copied Key ARN for the encrypted bucket. @@ -143,4 +143,4 @@ See the first video in the [managed credentials overview](../index.md) ## Next Steps -- [Enabling CORS in AWS S3](cors.md) +- [Enabling CORS in AWS S3](cors.md) \ No newline at end of file diff --git a/docs/docs/getting-started/storage-and-creds/managed-credentials/azure/cors.md b/docs/docs/getting-started/storage-and-creds/managed-credentials/azure/cors.md index 37a523b6d6..f474273248 100644 --- a/docs/docs/getting-started/storage-and-creds/managed-credentials/azure/cors.md +++ b/docs/docs/getting-started/storage-and-creds/managed-credentials/azure/cors.md @@ -15,11 +15,11 @@ Cross-Origin Resource Sharing (CORS) is typically enabled by default in Azure. I 3\. Open the `Resource sharing (CORS)` section on the left nav. -
+
4\. Add the following items to the permissions. -
+
| Allowed origins | Allowed methods | Allowed headers | | ------------------------- | --------------- | --------------- | @@ -27,4 +27,4 @@ Cross-Origin Resource Sharing (CORS) is typically enabled by default in Azure. I ## Next Steps -- [Provisioning Federated Credentials](provisioning.md) +- [Provisioning Federated Credentials](provisioning.md) \ No newline at end of file diff --git a/docs/docs/getting-started/storage-and-creds/managed-credentials/azure/provisioning.md b/docs/docs/getting-started/storage-and-creds/managed-credentials/azure/provisioning.md index 838fd7fddc..86d2cf64c9 100644 --- a/docs/docs/getting-started/storage-and-creds/managed-credentials/azure/provisioning.md +++ b/docs/docs/getting-started/storage-and-creds/managed-credentials/azure/provisioning.md @@ -17,23 +17,23 @@ The most secure method for connecting data from your Azure storage to Deep Lake 3\. Click on `Register an application` or `New registration`. -
+
4\. Enter the `Name` and `Supported account type` (all are supported in Deep Lake) and click `Register` -
+
5\. In the application console, click `Certificates & secrets`. -
+
6\. Click on `Federated credentials` and `Add credential`. -
+
7\. Click on `Select scenario` and select `Other issuer`. -
+
8\. Enter the following information in the form, and click `Add`. @@ -44,7 +44,7 @@ The most secure method for connecting data from your Azure storage to Deep Lake * **Description (optional):** enter description a with your choice * **Audience:** us-east-1:57e5de2f-e2ec-4514-b9b0-f3bb8c4283c3 -
+
#### Step 2a: Apply the Application Credentials to your Azure storage account @@ -54,23 +54,23 @@ The most secure method for connecting data from your Azure storage to Deep Lake 2\. Select the `Storage account` to which you want to add Application Credentials. -
+
3\. Select `Access Control (IAM)` and click `Add`, and `select Add role assignment`. -
+
4\. Search and select `Storage Blob Data Contributor` under the role names and click `Next`. -
+
5\. Click on the `Select members` link, and in the tab that opens up on the right, search by name and select the application you created in Step 1. Click `Select` at the bottom of the page. -
+
6\. The application should appear in the list of Members, at which point you can click `Review + assign`. -
+
#### Step 2b: Apply the Application Credentials to a specific Azure contained in your Azure storage account @@ -78,15 +78,15 @@ The most secure method for connecting data from your Azure storage to Deep Lake 2\. Select the `Storage account` to which you want to add Application Credentials. -
+
3\. Select the `Container` to which you add the Application Credentials. -
+
4\. Select `Access Control (IAM)` and click `Add`, and `select Add role assignment`. -
+
#### IMPORTANT TO PERFORM STEPS BELOW TO COMPLETE 2b - PLEASE DO NOT SKIP diff --git a/docs/docs/getting-started/storage-and-creds/managed-credentials/azure/workload-identities.md b/docs/docs/getting-started/storage-and-creds/managed-credentials/azure/workload-identities.md index a87cb2d49b..2bb86c2006 100644 --- a/docs/docs/getting-started/storage-and-creds/managed-credentials/azure/workload-identities.md +++ b/docs/docs/getting-started/storage-and-creds/managed-credentials/azure/workload-identities.md @@ -23,15 +23,15 @@ Set up a Workload Identity using the following steps: 1. Navigate to Managed Identities in Azure - + 1. Click `Create` a Managed Identity - + 1. Select the `Subscription` and `Resource Group` containing the workload, and give the Managed Identity a `Name`. Click `Review + Create`. - + ## Step 2: Attached the Azure Managed Identity to your workload @@ -40,17 +40,17 @@ When creating or updating a resource that will serve as the Client running Deep For example, in Azure Machine Learning Studio, when creating a compute instance, toggle `Assign Identity` and select the `Managed Identity` from Step 1. - + ## Step 3: Create a Deep Lake Workload Identity using the Azure Managed Identity Navigate to the `Permissions` tab for your organization in the [Deep Lake App](https://app.activeloop.ai/), locate the `Workload Identities`, and select `Add`. - + Specify a `Display Name`, `Client ID` (for the Managed Identity), and `Tenant ID`. The `Client ID` can be found in the main page for the Managed Identity, and the `Tenant ID` can be found in `Tenant Properties` in Azure. Click `Add`. - + ## Step 4: Run the workload @@ -76,4 +76,4 @@ Specifying the `AZURE_CLIENT_ID` is not necessary in some environments because t For a compute instance in the Azure Machine Learning Studio, the Client ID can be found in instance settings below: - + diff --git a/docs/docs/getting-started/storage-and-creds/managed-credentials/gcp/provisioning.md b/docs/docs/getting-started/storage-and-creds/managed-credentials/gcp/provisioning.md index fd81f476aa..2e480ed5a2 100644 --- a/docs/docs/getting-started/storage-and-creds/managed-credentials/gcp/provisioning.md +++ b/docs/docs/getting-started/storage-and-creds/managed-credentials/gcp/provisioning.md @@ -15,37 +15,37 @@ The most secure method for connecting data from your Google Cloud Storage bucket 2\. Navigate to `IAM & Admin` -> `Service Accounts` -> `CREATE SERVICE ACCOUNT` -
-
+
+
3\. Enter the `service account id`, and optional `name` and `description`. Make sure to copy the email address and and click on `CREATE AND CONTINUE`. -
+
4\. Click `CONTINUE` without entering any information. -
+
5\. Enter `activeloop-platform@activeloop-saas-iam.iam.gserviceaccount.com` in the `Service account users role` and click `DONE`. -
+
#### Step 2: Grant Access to the bucket using a Service Account Principal 1\. Navigate to `Cloud Storage` and `Buckets`. -
+
2\. Select `Edit Access` for the bucket you want to connect to Activeloop. -
+
3\. Select `Add Principal`. -
+
4\. Enter the `Service Account Email`, select the role as `Storage Object Admin`, and click `Save`. If the bucket is encrypted with customer managed KMS key, then `Cloud KMS CryptoKey Encrypter/Decrypter` should be added in the `Role` field as well. -
+
#### Step 3: Enter the Service Account Email (Step 2) into the Activeloop App @@ -53,4 +53,4 @@ See the first video in the [managed credentials overview](../index.md) ## Next Steps -- [Enabling CORS in GCP](cors.md) +- [Enabling CORS in GCP](cors.md) \ No newline at end of file diff --git a/docs/docs/guide/deep-learning/deep-learning.ipynb b/docs/docs/guide/deep-learning/deep-learning.ipynb index caec3a728f..f2b8a0c7ef 100644 --- a/docs/docs/guide/deep-learning/deep-learning.ipynb +++ b/docs/docs/guide/deep-learning/deep-learning.ipynb @@ -77,9 +77,27 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 4, "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Dataset(columns=(images,labels,boxes), length=6471)\n", + "\u001B[1m+------+--------------------------------------------+\n", + "|column| type |\n", + "+------+--------------------------------------------+\u001B[0m\n", + "|images| array(dtype=uint8, shape=[None,None,None]) |\n", + "+------+--------------------------------------------+\n", + "|labels| array(dtype=uint32, shape=[None]) |\n", + "+------+--------------------------------------------+\n", + "|boxes |array(dtype=float32, shape=[None,None,None])|\n", + "+------+--------------------------------------------+\n", + "\n" + ] + } + ], "source": [ "ds.summary()" ] @@ -141,9 +159,19 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 57, "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " % Total % Received % Xferd Average Speed Time Time Time Current\n", + " Dload Upload Total Spent Left Speed\n", + "100 167k 0 167k 0 0 426k 0 --:--:-- --:--:-- --:--:-- 425k\n" + ] + } + ], "source": [ "# Download dataset\n", "from IPython.display import clear_output\n", @@ -213,12 +241,12 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 59, "metadata": {}, "outputs": [], "source": [ "# Find the class_names and list of files that need to be uploaded\n", - "dataset_folder = \"animals\"\n", + "dataset_folder = 'animals'\n", "\n", "# Find the subfolders, but filter additional files like DS_Store that are added on Mac machines.\n", "class_names = [item for item in os.listdir(dataset_folder) if os.path.isdir(os.path.join(dataset_folder, item))]\n", @@ -256,9 +284,25 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 68, "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Dataset(columns=(images,labels), length=0)\n", + "\u001B[1m+------+------------------------------------------+\n", + "|column| type |\n", + "+------+------------------------------------------+\u001B[0m\n", + "|images|array(dtype=uint8, shape=[None,None,None])|\n", + "+------+------------------------------------------+\n", + "|labels| array(dtype=uint32, shape=[None]) |\n", + "+------+------------------------------------------+\n", + "\n" + ] + } + ], "source": [ "ds.summary()" ] diff --git a/docs/docs/guide/rag.md b/docs/docs/guide/rag.md index a2b935c8bd..c04add25cd 100644 --- a/docs/docs/guide/rag.md +++ b/docs/docs/guide/rag.md @@ -689,9 +689,19 @@ results = fusion(docs_vs, docs_bm25) print(results) ``` -The exact scores depend on the retrieved rows; use the sorted results below. - - +Output: +``` +{'2637': Document(id='2637', data={'restaurant_name': 'Mifen101 花溪米粉王', 'restaurant_review': 'Feel like I’m back in China.'}, score=0.054989174038501676), + '5136': Document(id='5136', data={'restaurant_name': 'Scratch', 'restaurant_review': 'Just had drinks. They were good!'}, score=0.09802189349997713), + '17426': Document(id='17426', data={'restaurant_name': "St. Stephen's Green", 'restaurant_review': 'Good drinks an easy going bartenders'}, score=0.09831671336973409), + '17444': Document(id='17444', data={'restaurant_name': "St. Stephen's Green", 'restaurant_review': 'Good drinks, good food'}, score=0.09900385707681324), + '2496': Document(id='2496', data={'restaurant_name': 'Seasons Noodles & Dumplings Garden', 'restaurant_review': 'Comfort food, excellent service! Feel like back to home.'}, score=0.02172369191268401), + '4022': Document(id='4022', data={'restaurant_name': 'Eureka! Mountain View', 'restaurant_review': 'Good drinks and burgers'}, score=0.0985337276269904), + '3518': Document(id='3518', data={'restaurant_name': 'Olympus Caffe & Bakery', 'restaurant_review': 'I like the garden to sit down with friends and have a drink.'}, score=0.3566115095933449), + '17502': Document(id='17502', data={'restaurant_name': "St. Stephen's Green", 'restaurant_review': 'Nice place for a drink'}, score=0.10612380842648524), + '11383': Document(id='11383', data={'restaurant_name': 'Ludwigs Biergarten Mountain View', 'restaurant_review': 'Beer is fresh tables are big feel like a proper beer garden'}, score=0.045790150271479965), + '10788': Document(id='10788', data={'restaurant_name': 'Casa Lupe', 'restaurant_review': 'Run by a family that makes you feel like part of the family. Awesome food. I love their wet Chili Verde burritos'}, score=0.02088547418398944)} +``` We sort the results dictionary by each document's combined score in descending order, ensuring that the highest-ranking documents appear first. @@ -700,7 +710,19 @@ sorted_documents = dict(sorted(results.items(), key=lambda item: item[1].score, print(sorted_documents) ``` -The sorted output contains the retrieved documents in descending fused score order. +Output: +``` +{'3518': Document(id='3518', data={'restaurant_name': 'Olympus Caffe & Bakery', 'restaurant_review': 'I like the garden to sit down with friends and have a drink.'}, score=0.3566115095933449), + '17502': Document(id='17502', data={'restaurant_name': "St. Stephen's Green", 'restaurant_review': 'Nice place for a drink'}, score=0.10612380842648524), + '17444': Document(id='17444', data={'restaurant_name': "St. Stephen's Green", 'restaurant_review': 'Good drinks, good food'}, score=0.09900385707681324), + '4022': Document(id='4022', data={'restaurant_name': 'Eureka! Mountain View', 'restaurant_review': 'Good drinks and burgers'}, score=0.0985337276269904), + '17426': Document(id='17426', data={'restaurant_name': "St. Stephen's Green", 'restaurant_review': 'Good drinks an easy going bartenders'}, score=0.09831671336973409), + '5136': Document(id='5136', data={'restaurant_name': 'Scratch', 'restaurant_review': 'Just had drinks. They were good!'}, score=0.09802189349997713), + '2637': Document(id='2637', data={'restaurant_name': 'Mifen101 花溪米粉王', 'restaurant_review': 'Feel like I’m back in China.'}, score=0.054989174038501676), + '11383': Document(id='11383', data={'restaurant_name': 'Ludwigs Biergarten Mountain View', 'restaurant_review': 'Beer is fresh tables are big feel like a proper beer garden'}, score=0.045790150271479965), + '2496': Document(id='2496', data={'restaurant_name': 'Seasons Noodles & Dumplings Garden', 'restaurant_review': 'Comfort food, excellent service! Feel like back to home.'}, score=0.02172369191268401), + '10788': Document(id='10788', data={'restaurant_name': 'Casa Lupe', 'restaurant_review': 'Run by a family that makes you feel like part of the family. Awesome food. I love their wet Chili Verde burritos'}, score=0.02088547418398944)} +``` ### Show the results @@ -842,8 +864,8 @@ for v in sorted_documents.values(): print(f"Restaurant name: {v.data['restaurant_name']} \nReview: {v.data['restaurant_review']}") ``` -Example output from one run (your rows may differ if either dataset changes): - +Output: +``` Restaurant name: Olympus Caffe & Bakery Review: I like the garden to sit down with friends and have a drink. Restaurant name: St. Stephen's Green @@ -1266,7 +1288,7 @@ Dataset(columns=(text,embedding), length=19719) ``` ```python -# The copied dataset already has an `embedding` column, so reuse it. +# The copied dataset already has an embedding column, so reuse it. ``` 3. **Text Extraction** : diff --git a/docs/docs/llms.txt b/docs/docs/llms.txt index 1fe530a5ec..ad5a57fa22 100644 --- a/docs/docs/llms.txt +++ b/docs/docs/llms.txt @@ -194,4 +194,4 @@ cats = view.query("SELECT * WHERE species = 'cat'") - [Column API](https://docs.deeplake.ai/api/column/): Column management and indexing - [Query API](https://docs.deeplake.ai/api/query/): TQL syntax and examples - [Data Types](https://docs.deeplake.ai/api/types/): All supported data types -- [TQL Reference](https://docs.deeplake.ai/advanced/tql/): Complete TQL syntax guide +- [TQL Reference](https://docs.deeplake.ai/advanced/tql/): Complete TQL syntax guide \ No newline at end of file