Skip to content

Data lake storage support phase2 #1934

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Mahmoud Bahaa (mahmoudbahaa) wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Data lake storage support phase2 #1934

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
5f3b446
* Added "@loki @Sql" to all test cases except 2
mahmoudbahaa Apr 22, 2023
4613bc2
* Full support for sql for the same features supported by Loki
mahmoudbahaa Apr 22, 2023
4e363b2
* Minimal changes for blob end point to allow for extension in dfs en…
mahmoudbahaa Apr 22, 2023
a1ada62
* Introduce new end point dfs and added new test cases for it
mahmoudbahaa Apr 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 71 additions & 0 deletions .vscode/launch.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,30 @@
},
"outputCapture": "std"
},
{
"type": "node",
"request": "launch",
"name": "Azurite Blob Service - Loki",
"cwd": "${workspaceFolder}",
"runtimeArgs": ["-r", "ts-node/register"],
"args": ["${workspaceFolder}/src/blob/main.ts", "-d", "debug.log", "--skipApiVersionCheck"],
"env": {
"AZURITE_ACCOUNTS": ""
},
"outputCapture": "std"
},
{
"type": "node",
"request": "launch",
"name": "Azurite DataLake Service - Loki",
"cwd": "${workspaceFolder}",
"runtimeArgs": ["-r", "ts-node/register"],
"args": ["${workspaceFolder}/src/dfs/main.ts", "-d", "debug.log", "--skipApiVersionCheck"],
"env": {
"AZURITE_ACCOUNTS": ""
},
"outputCapture": "std"
},
{
"type": "node",
"request": "launch",
Expand Down Expand Up @@ -204,6 +228,53 @@
"internalConsoleOptions": "openOnSessionStart",
"outputCapture": "std"
},
{
"type": "node",
"request": "launch",
"name": "Current Mocha TS File - DataLake SQL",
"cwd": "${workspaceFolder}",
"runtimeArgs": ["-r", "ts-node/register"],
"args": [
"${workspaceFolder}/node_modules/mocha/bin/_mocha",
"-u",
"tdd",
"--timeout",
"999999",
"--colors",
"${workspaceFolder}/${relativeFile}"
],
"env": {
"AZURITE_ACCOUNTS": "",
"AZURITE_TEST_DB": "mysql://root:my-secret-pw@127.0.0.1:3306/azurite_blob_test",
"NODE_TLS_REJECT_UNAUTHORIZED": "0",
"IS_DATALAKE": "true",
},
"internalConsoleOptions": "openOnSessionStart",
"outputCapture": "std"
},
{
"type": "node",
"request": "launch",
"name": "Current Mocha TS File - PostgreSQL",
"cwd": "${workspaceFolder}",
"runtimeArgs": ["-r", "ts-node/register"],
"args": [
"${workspaceFolder}/node_modules/mocha/bin/_mocha",
"-u",
"tdd",
"--timeout",
"999999",
"--colors",
"${workspaceFolder}/${relativeFile}"
],
"env": {
"AZURITE_ACCOUNTS": "",
"AZURITE_TEST_DB": "postgres://postgres:postgres@127.0.0.1:5432/azurite_dfs_test",
"NODE_TLS_REJECT_UNAUTHORIZED": "0"
},
"internalConsoleOptions": "openOnSessionStart",
"outputCapture": "std"
},
{
"name": "VSC Extension",
"type": "extensionHost",
Expand Down
5 changes: 5 additions & 0 deletions ChangeLog.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,11 @@ Blob:

- Fixed issue of: blob batch subresponse is slightly different from the on from Azure serivce, which causes exception in CPP SDK.
- Fixed issue of: setMetadata API allows invalid metadata name with hyphen.
- Support Same features in SQL Metadata Store as Loki Metadata Store (support Blob Copy & Page Blob)

DataLake:

- Introduced dfs endpoint with the following features see readme for more details on what is supported and what is not

## 2023.03 Version 3.23.0

Expand Down
21 changes: 16 additions & 5 deletions README.mcr.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,37 +17,44 @@ Azurite is an open source Azure Storage API compatible server (emulator). Based
# How to Use this Image

```bash
docker run -p 10000:10000 -p 10001:10001 -p 10002:10002 mcr.microsoft.com/azure-storage/azurite
docker run -p 10000:10000 -p 10001:10001 -p 10002:10002 -p 10003:10003 mcr.microsoft.com/azure-storage/azurite
```

`-p 10000:10000` will expose blob service's default listening port.
`-p 10001:10001` will expose queue service's default listening port.
`-p 10002:10002` will expose table service's default listening port.
`-p 10003:10003` will expose datalake service's default listening port.

Just run blob service:

```bash
docker run -p 10000:10000 mcr.microsoft.com/azure-storage/azurite azurite-blob --blobHost 0.0.0.0
```

Just run datalake service:

```bash
docker run -p 10000:10000 mcr.microsoft.com/azure-storage/azurite azurite-datalake --datalakeHost 0.0.0.0
```

Run the image as a service (`-d` = deamon) named `azurite` and restart unless specifically stopped (this is useful when re-starting your development machine for example)

```bash
docker run --name azurite -d --restart unless-stopped -p 10000:10000 -p 10001:10001 -p 10002:10002 mcr.microsoft.com/azure-storage/azurite
docker run --name azurite -d --restart unless-stopped -p 10000:10000 -p 10001:10001 -p 10002:10002 -p 10003:10003 mcr.microsoft.com/azure-storage/azurite
```

**Run Azurite V3 docker image with customized persisted data location**

```bash
docker run -p 10000:10000 -p 10001:10001 -p 10002:10002 -v c:/azurite:/data mcr.microsoft.com/azure-storage/azurite
docker run -p 10000:10000 -p 10001:10001 -p 10002:10002 -p 10003:10003 -v c:/azurite:/data mcr.microsoft.com/azure-storage/azurite
```

`-v c:/azurite:/data` will use and map host path `c:/azurite` as Azurite's workspace location.

**Customize Azurite V3 supported parameters for docker image**

```bash
docker run -p 7777:7777 -p 8888:8888 -p 9999:9999 -v c:/azurite:/workspace mcr.microsoft.com/azure-storage/azurite azurite -l /workspace -d /workspace/debug.log --blobPort 7777 --blobHost 0.0.0.0 --queuePort 8888 --queueHost 0.0.0.0 --tablePort 9999 --tableHost 0.0.0.0 --loose --skipApiVersionCheck --disableProductStyleUrl
docker run -p 7777:7777 -p 8888:8888 -p 9999:9999 -p 6666:6666 -v c:/azurite:/workspace mcr.microsoft.com/azure-storage/azurite azurite -l /workspace -d /workspace/debug.log --blobPort 7777 --blobHost 0.0.0.0 --queuePort 8888 --queueHost 0.0.0.0 --tablePort 9999 --tableHost 0.0.0.0--datalakePort 6666 --datalakeHost 0.0.0.0 --loose --skipApiVersionCheck --disableProductStyleUrl
```

Above command will try to start Azurite image with configurations:
Expand All @@ -68,13 +75,17 @@ Above command will try to start Azurite image with configurations:

`--tableHost 0.0.0.0` defines table service listening endpoint to accept requests from host machine.

`--datalakePort 6666` makes Azurite blob service listen to port 6666, while `-p 6666:6666` redirects requests from host machine's port 6666 to docker instance.

`--datalakeHost 0.0.0.0` defines blob service listening endpoint to accept requests from host machine.

`--loose` enables loose mode which ignore unsupported headers and parameters.

`--skipApiVersionCheck` skip the request API version check.

`--disableProductStyleUrl` force parsing storage account name from request Uri path, instead of from request Uri host.

> If you use customized azurite paramters for docker image, `--blobHost 0.0.0.0`, `--queueHost 0.0.0.0` are required parameters.
> If you use customized azurite paramters for docker image, `--blobHost 0.0.0.0`, `--datalakeHost 0.0.0.0`, `--queueHost 0.0.0.0` are required parameters.

> In above sample, you need to use **double first forward slash** for location and debug path parameters to avoid a [known issue](https://stackoverflow.com/questions/48427366/docker-build-command-add-c-program-files-git-to-the-path-passed-as-build-argu) for Git on Windows.

Expand Down
Loading