Skip to content
Open
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions docs/docs/advanced/best-practices.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ The second method is more efficient as the Deep Lake columnar format is handling

### Avoid decompressing images when adding them to the dataset

Deep Lake [Image](../../api/types/#deeplake.types.Image) type supports multiple image formats including JPEG, PNG, and TIFF. When adding images to the dataset, no need to decompress images into `numpy.array` or `PIL.Image` objects. Instead you can pass raw bytes and Deep Lake will automatically decompress the images when reading them.
Deep Lake [Image](../api/types/#deeplake.types.Image) type supports multiple image formats including JPEG, PNG, and TIFF. When adding images to the dataset, no need to decompress images into `numpy.array` or `PIL.Image` objects. Instead you can pass raw bytes and Deep Lake will automatically decompress the images when reading them.

```python
image_ds.add_column("images", deeplake.types.Image("uint8", "jpeg"))
Expand All @@ -88,9 +88,9 @@ image_ds.append({"images": f.read()})

This will be much faster and use less memory compared to decompressing the images before adding them to the dataset.

### Use [Link Image](../../api/types/#deeplake.types.Link) for cloud storage images
### Use [Link Image](../api/types/#deeplake.types.Link) for cloud storage images

If the images are in the cloud storage and you want to avoid data duplication you can use [Link Image](../../api/types/#deeplake.types.Link) type. In this case you need to pass the image url.
If the images are in the cloud storage and you want to avoid data duplication you can use [Link Image](../api/types/#deeplake.types.Link) type. In this case you need to pass the image url.

```python
link_image_ds.add_column("images", deeplake.types.Link(deeplake.types.Image()))
Expand All @@ -111,7 +111,7 @@ Small data updates like `ds['column1'][i] = value` can be done in Deep Lake data

### Prefer opening datasets in read-only mode.

When you are not planning to modify the dataset, prefer opening the dataset with [open_read_only](../../api/dataset/#deeplake.open_read_only) method. This will prevent accidental modifications to the dataset and will improve the performance of the data access.
When you are not planning to modify the dataset, prefer opening the dataset with [open_read_only](../api/dataset/#deeplake.open_read_only) method. This will prevent accidental modifications to the dataset and will improve the performance of the data access.

### Prefer batch access instead of row by row access

Expand Down Expand Up @@ -139,15 +139,15 @@ for i in range(len(column)):

### Use `query` for complex data filtering and search

Deep Lake supports SQL-like queries to filter and search the data. If you need to filter the data based on multiple columns or complex conditions, prefer using the [deeplake.DatasetView.query](../../api/dataset#deeplake.DatasetView) or [deeplake.query](../../api/query) method, instead of doing manual iteration and filtering.
Deep Lake supports SQL-like queries to filter and search the data. If you need to filter the data based on multiple columns or complex conditions, prefer using the [deeplake.DatasetView.query](../api/dataset#deeplake.DatasetView) or [deeplake.query](../api/query) method, instead of doing manual iteration and filtering.

### Avoid accessing the data of the whole column

The column data can be accessed directly by `ds["column_name"][:]`. For the large datasets this can lead to memory issues. Prefer divide the data into batches and process them separately.

### Consider using async data access

Deep Lake supports async data access and query. If your workflow allows async processing and benefits from that, consider using async data access. Please refer to the [Async Data Loader](../../guide/deep-learning/async-data-loader) guide for the details.
Deep Lake supports async data access and query. If your workflow allows async processing and benefits from that, consider using async data access. Please refer to the [Async Data Loader](../guide/deep-learning/async-data-loader) guide for the details.


## Storage and Data Management
Expand All @@ -169,15 +169,15 @@ Deep Lake supports multiple storage backends, which are differentiated by url sc

In memory datasets are not persisted and are useful for temporary data storage and testing. They are automatically deleted when the process is terminated.

If you created in memory dataset and you want to persist it you can use [deeplake.copy](../../advanced/sync/#copying-datasets) method to copy the dataset to the local or cloud storage.
If you created in memory dataset and you want to persist it you can use [deeplake.copy](../advanced/sync/#copying-datasets) method to copy the dataset to the local or cloud storage.

### Avoid local storage for large datasets

Local storage shows better latency compared to the cloud dataset. However it is not recommended to use local storage for large dataset as with the scale the performance can be degraded.

If you plan to store large data (>=100GB) cloud storage will be more efficient and reliable.

If you have local dataset and you want to move it to the cloud you can use [deeplake.copy](../../advanced/sync/#copying-datasets) method to copy the dataset to the cloud storage.
If you have local dataset and you want to move it to the cloud you can use [deeplake.copy](../advanced/sync/#copying-datasets) method to copy the dataset to the cloud storage.

### Prefer accessing the cloud storage from the same region

Expand Down
2 changes: 1 addition & 1 deletion docs/docs/advanced/visualize.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ description: "Learn how to visualize Deep Lake datasets using the Deep Lake web

# Dataset Visualization

Deep Lake has a web interface for visualizing, versioning, and querying machine learning datasets. It utilizes the Deep Lake format under-the-hood, and it can be connected to datasets stored in all Deep Lake [storage locations](../../getting-started/storage-and-creds/).
Deep Lake has a web interface for visualizing, versioning, and querying machine learning datasets. It utilizes the Deep Lake format under-the-hood, and it can be connected to datasets stored in all Deep Lake [storage locations](../getting-started/storage-and-creds/).

![Visualizer](images/visualize.jpg)

Expand Down
1 change: 0 additions & 1 deletion docs/docs/api/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,6 @@ This reference documents the Python API of Deep Lake.
- [Query Language](query.md): Complete TQL syntax and operations
- [Metadata](metadata.md): Dataset and column metadata management
- [Version Control](version_control.md): Version control, history, branches, and tags
- [Miscellaneous](misc.md): Additional auxiliary functionality

## Index Management

Expand Down
2 changes: 1 addition & 1 deletion docs/docs/api/query.md
Original file line number Diff line number Diff line change
Expand Up @@ -229,7 +229,7 @@ results = deeplake.query("""
l.label
FROM "s3://bucket1/images" AS i
JOIN "gcs://bucket2/embeddings" AS e ON i.id = e.image_id
JOIN "azure://container/labels" AS l ON i.id = l.image_id
JOIN "azure://<account_name>/<container_name>/labels" AS l ON i.id = l.image_id
WHERE l.confidence > 0.9
ORDER BY COSINE_SIMILARITY(e.embedding, ARRAY[0.1, 0.2, 0.3]) DESC
LIMIT 100
Expand Down
4 changes: 0 additions & 4 deletions docs/docs/concepts/.pages

This file was deleted.

6 changes: 3 additions & 3 deletions docs/docs/getting-started/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ ds = deeplake.create("path/to/dataset")
# Or create in cloud storage
ds = deeplake.create("s3://my-bucket/dataset")
ds = deeplake.create("gcs://my-bucket/dataset")
ds = deeplake.create("azure://container/dataset")
ds = deeplake.create("azure://<account_name>/<container_name>/<dataset_name>")
```

## Adding Data
Expand Down Expand Up @@ -179,8 +179,8 @@ cats = future.result()

## Next Steps

- Explore [RAG applications](../../guide/rag)
- Check out [Deep Learning integration](../../guide/deep-learning/deep-learning)
- Explore [RAG applications](../guide/rag.md)
- Check out [Deep Learning integration](../guide/deep-learning/deep-learning.ipynb)

## Support

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ In order to visualize Deep Lake datasets stored in your own S3 buckets in the [D
"HEAD"
],
"AllowedOrigins": [
"*.activeloop.ai"
"https://*.activeloop.ai"
],
"ExposeHeaders": []
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,11 @@ The most secure method for connecting data from your AWS account to Deep Lake is

3\. In the left nav, open the `Policies` under `Access management` and on `Create policy` on the right.

<figure><img src="../images/IAM_Provisioning_Screenshots.001.jpeg" alt=""><figcaption></figcaption></figure>
<figure><img src="images/IAM_Provisioning_Screenshots.001.jpeg" alt=""><figcaption></figcaption></figure>

5\. Select the `JSON` tab instead of `Visual editor`.

<figure><img src="../images/IAM_Provisioning_Screenshots.002.jpeg" alt=""><figcaption></figcaption></figure>
<figure><img src="images/IAM_Provisioning_Screenshots.002.jpeg" alt=""><figcaption></figcaption></figure>

6\. Replace the code in the editor with the code below. Replace `BUCKET_NAME` with the bucket names for which you want to grant role-based access:

Expand Down Expand Up @@ -50,17 +50,17 @@ The most secure method for connecting data from your AWS account to Deep Lake is

7\. On the bottom right, click `Next: Tags` (create tags if needed) and `Next: Preview`, enter the policy `name` and `description`, and click `Create policy`

<figure><img src="../images/IAM_Provisioning_Screenshots.003.jpeg" alt=""><figcaption></figcaption></figure>
<figure><img src="images/IAM_Provisioning_Screenshots.003.jpeg" alt=""><figcaption></figcaption></figure>

#### Step 2: Create the AWS IAM Role&#x20;

1\. On the `IAM` page, in the left nav, open the `Roles` under `Access management`, and click `Create role` on the right.

<figure><img src="../images/IAM_Provisioning_Screenshots.004.jpeg" alt=""><figcaption></figcaption></figure>
<figure><img src="images/IAM_Provisioning_Screenshots.004.jpeg" alt=""><figcaption></figcaption></figure>

3\. Select `Custom trust policy` from the list of options.

<figure><img src="../images/IAM_Provisioning_Screenshots.005.jpeg" alt=""><figcaption></figcaption></figure>
<figure><img src="images/IAM_Provisioning_Screenshots.005.jpeg" alt=""><figcaption></figcaption></figure>

4\. Replace the policy definition with the code below and click `Next`

Expand All @@ -83,23 +83,23 @@ The most secure method for connecting data from your AWS account to Deep Lake is

5\. From the provided policy list, select the previously created policy from Step 1 and click `Next`

<figure><img src="../images/IAM_Provisioning_Screenshots.010.jpeg" alt=""><figcaption></figcaption></figure>
<figure><img src="images/IAM_Provisioning_Screenshots.010.jpeg" alt=""><figcaption></figcaption></figure>

6\. Set the `name` and `description` for the role and click `Create role` at the bottom.

<figure><img src="../images/IAM_Provisioning_Screenshots.007.jpeg" alt=""><figcaption></figcaption></figure>
<figure><img src="images/IAM_Provisioning_Screenshots.007.jpeg" alt=""><figcaption></figcaption></figure>

#### Step 3: Grant Access to AWS KMS Key (**only for buckets that are encrypted with customer managed KMS keys**)

1\. Navigate to the bucket in the AWS S3 UI

2\. Open the bucket Properties

<figure><img src="../images/IAM_Provisioning_Screenshots.008.jpeg" alt=""><figcaption></figcaption></figure>
<figure><img src="images/IAM_Provisioning_Screenshots.008.jpeg" alt=""><figcaption></figcaption></figure>

3\. Scroll down to Default encryption and copy the `AWS KMS key ARN`&#x20;

<figure><img src="../images/IAM_Provisioning_Screenshots.009.jpeg" alt=""><figcaption></figcaption></figure>
<figure><img src="images/IAM_Provisioning_Screenshots.009.jpeg" alt=""><figcaption></figcaption></figure>

4\. In the Policy creation step (Step 1, Sub-step 6), use the JSON below in the policy statement, and replace `YOUR_KMS_KEY_ARN` with the copied Key ARN for the encrypted bucket.

Expand Down Expand Up @@ -129,7 +129,7 @@ The most secure method for connecting data from your AWS account to Deep Lake is
"kms:DescribeKey"
],
"Resource": [
"YOUR_KMS_KEY_ARN”
"YOUR_KMS_KEY_ARN"
]
}
]
Expand All @@ -143,4 +143,4 @@ See the first video in the [managed credentials overview](../index.md)

## Next Steps

- [Enabling CORS in AWS S3](cors.md)
- [Enabling CORS in AWS S3](cors.md)
Original file line number Diff line number Diff line change
Expand Up @@ -19,12 +19,12 @@ Cross-Origin Resource Sharing (CORS) is typically enabled by default in Azure. I

4\. Add the following items to the permissions.

<figure><img src="images/Azure_CORS_1.png" alt=""><figcaption></figcaption></figure>
<figure><img src="images/Azure_CORS_2.png" alt=""><figcaption></figcaption></figure>

| Allowed origins | Allowed methods | Allowed headers |
| ------------------------- | --------------- | --------------- |
| https://app.activeloop.ai | GET, HEAD | \* |

## Next Steps

- [Provisioning Federated Credentials](provisioning.md)
- [Provisioning Federated Credentials](provisioning.md)
Original file line number Diff line number Diff line change
Expand Up @@ -17,23 +17,23 @@ The most secure method for connecting data from your Azure storage to Deep Lake

3\. Click on `Register an application` or `New registration`.

<figure><img src="../images/Azure_Federated_Step_1_3.png" alt=""><figcaption></figcaption></figure>
<figure><img src="images/Azure_Federated_Step_1_3.png" alt=""><figcaption></figcaption></figure>

4\. Enter the `Name` and `Supported account type` (all are supported in Deep Lake) and click `Register`

<figure><img src="../images/Azure_Federated_Step_1_4.png" alt=""><figcaption></figcaption></figure>
<figure><img src="images/Azure_Federated_Step_1_4.png" alt=""><figcaption></figcaption></figure>

5\. In the application console, click `Certificates & secrets`.

<figure><img src="../images/Azure_Federated_Step_1_5.png" alt=""><figcaption></figcaption></figure>
<figure><img src="images/Azure_Federated_Step_1_5.png" alt=""><figcaption></figcaption></figure>

6\. Click on `Federated credentials` and `Add credential`.

<figure><img src="../images/Azure_Federated_Step_1_6.png" alt=""><figcaption></figcaption></figure>
<figure><img src="images/Azure_Federated_Step_1_6.png" alt=""><figcaption></figcaption></figure>

7\. Click on `Select scenario` and select `Other issuer`.

<figure><img src="../images/Azure_Federated_Step_1_7.png" alt=""><figcaption></figcaption></figure>
<figure><img src="images/Azure_Federated_Step_1_7.png" alt=""><figcaption></figcaption></figure>

8\. Enter the following information in the form, and click `Add`.

Expand All @@ -44,7 +44,7 @@ The most secure method for connecting data from your Azure storage to Deep Lake
* **Description (optional):** enter description a with your choice
* **Audience:** us-east-1:57e5de2f-e2ec-4514-b9b0-f3bb8c4283c3

<figure><img src="../images/Azure_Federated_Step_1_8.png" alt=""><figcaption></figcaption></figure>
<figure><img src="images/Azure_Federated_Step_1_8.png" alt=""><figcaption></figcaption></figure>

#### Step 2a: Apply the Application Credentials to your Azure storage account&#x20;

Expand All @@ -54,39 +54,39 @@ The most secure method for connecting data from your Azure storage to Deep Lake

2\. Select the `Storage account` to which you want to add Application Credentials.

<figure><img src="../images/Azure_Federated_Step_2a_2.png" alt=""><figcaption></figcaption></figure>
<figure><img src="images/Azure_Federated_Step_2a_2.png" alt=""><figcaption></figcaption></figure>

3\. Select `Access Control (IAM)` and click `Add`, and `select Add role assignment`.

<figure><img src="../images/Azure_Federated_Step_2a_3.png" alt=""><figcaption></figcaption></figure>
<figure><img src="images/Azure_Federated_Step_2a_3.png" alt=""><figcaption></figcaption></figure>

4\. Search and select `Storage Blob Data Contributor` under the role names and click `Next`.

<figure><img src="../images/Azure_Federated_Step_2a_4.png" alt=""><figcaption></figcaption></figure>
<figure><img src="images/Azure_Federated_Step_2a_4.png" alt=""><figcaption></figcaption></figure>

5\. Click on the `Select members` link, and in the tab that opens up on the right, search by name and select the application you created in Step 1. Click `Select` at the bottom of the page.

<figure><img src="../images/Azure_Federated_Step_2a_5.png" alt=""><figcaption></figcaption></figure>
<figure><img src="images/Azure_Federated_Step_2a_5.png" alt=""><figcaption></figcaption></figure>

6\. The application should appear in the list of Members, at which point you can click `Review + assign`.

<figure><img src="../images/Azure_Federated_Step_2a_6.png" alt=""><figcaption></figcaption></figure>
<figure><img src="images/Azure_Federated_Step_2a_6.png" alt=""><figcaption></figcaption></figure>

#### Step 2b: Apply the Application Credentials to a specific Azure contained in your Azure storage account

1\. Go to the `Storage accounts` page in the Azure UI, which can be done by searching "Storage accounts" in the console.

2\. Select the `Storage account` to which you want to add Application Credentials.

<figure><img src="../images/Azure_Federated_Step_2b_2.png" alt=""><figcaption></figcaption></figure>
<figure><img src="images/Azure_Federated_Step_2b_2.png" alt=""><figcaption></figcaption></figure>

3\. Select the `Container` to which you add the Application Credentials.

<figure><img src="../images/Azure_Federated_Step_2b_3.png" alt=""><figcaption></figcaption></figure>
<figure><img src="images/Azure_Federated_Step_2b_3.png" alt=""><figcaption></figcaption></figure>

4\. Select `Access Control (IAM)` and click `Add`, and `select Add role assignment`.

<figure><img src="../images/Azure_Federated_Step_2b_4.png" alt=""><figcaption></figcaption></figure>
<figure><img src="images/Azure_Federated_Step_2b_4.png" alt=""><figcaption></figcaption></figure>

#### IMPORTANT TO PERFORM STEPS BELOW TO COMPLETE 2b - PLEASE DO NOT SKIP

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,15 +23,15 @@ Set up a Workload Identity using the following steps:

1. Navigate to Managed Identities in Azure

<img src="../images/workload_step_1_1.png" class="screenshot">
<img src="images/workload_step_1_1.png" class="screenshot">

1. Click `Create` a Managed Identity

<img src="../images/workload_step_1_2.png" class="screenshot">
<img src="images/workload_step_1_2.png" class="screenshot">

1. Select the `Subscription` and `Resource Group` containing the workload, and give the Managed Identity a `Name`. Click `Review + Create`.

<img src="../images/workload_step_1_3.png" class="screenshot">
<img src="images/workload_step_1_3.png" class="screenshot">


## Step 2: Attached the Azure Managed Identity to your workload
Expand All @@ -40,17 +40,17 @@ When creating or updating a resource that will serve as the Client running Deep

For example, in Azure Machine Learning Studio, when creating a compute instance, toggle `Assign Identity` and select the `Managed Identity` from Step 1.

<img src="../images/workload_step_2.png" class="screenshot">
<img src="images/workload_step_2.png" class="screenshot">

## Step 3: Create a Deep Lake Workload Identity using the Azure Managed Identity

Navigate to the `Permissions` tab for your organization in the [Deep Lake App](https://app.activeloop.ai/), locate the `Workload Identities`, and select `Add`.

<img src="../images/workload_step_3_1.png" class="screenshot">
<img src="images/workload_step_3_1.png" class="screenshot">

Specify a `Display Name`, `Client ID` (for the Managed Identity), and `Tenant ID`. The `Client ID` can be found in the main page for the Managed Identity, and the `Tenant ID` can be found in `Tenant Properties` in Azure. Click `Add`.

<img src="../images/workload_step_3_2.png" class="screenshot">
<img src="images/workload_step_3_2.png" class="screenshot">

## Step 4: Run the workload

Expand All @@ -76,4 +76,4 @@ Specifying the `AZURE_CLIENT_ID` is not necessary in some environments because t

For a compute instance in the Azure Machine Learning Studio, the Client ID can be found in instance settings below:

<img src="../images/workload_step_4.png" class="screenshot">
<img src="images/workload_step_4.png" class="screenshot">
Loading