Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: azure blob storage support #883

Merged
merged 53 commits into from
Sep 9, 2024
Merged

feat: azure blob storage support #883

merged 53 commits into from
Sep 9, 2024

Conversation

khyurri
Copy link
Contributor

@khyurri khyurri commented Jul 18, 2024 •
edited
Loading

This PR integrates Azure Blob Storage into BadgerDoc and eliminates the use of boto3, minio, and aioboto3 across all microservices. Additionally, it standardizes the storage configuration.

Migration from previous version

Backward Compatibility

  1. In the previous version, when a signed url was passed from BadgerDoc to Airflow/Databricks, the parameter was named s3_signed_url. The current version renames this parameter to signed_url by default. However, to maintain backward compatibility, the parameter JOBS_SIGNER_URL_KEY_NAME is used to rename the signed URL key in the arguments passed.
    For example, setting JOBS_SIGNER_URL_KEY_NAME=s3_signed_url retains the original parameter name.
  2. The parameter S3_PRE_SIGNED_EXPIRES_HOURS has been renamed to JOBS_SIGNED_URL_TTL, and its values are now set in minutes rather than hours.

.env migration

  1. Rename S3_PRE_SIGNED_EXPIRES_HOURS -> JOBS_SIGNED_URL_TTL
  2. Rename JOBS_RUN_PIPELINES_WITH_SIGNED_URL -> JOBS_SIGNED_URL_ENABLED

files migration from Minio / S3 into Azure Blob Storage

-TBC-

Removed microservices

  • Remove convert microservice

Removed or skipped tests

-TBC-

Known issues

  • Python3.12 base image can't be built
  • Clear build dir after building Python3.12 image
  • Azure and Minio installations should automatically create containers or buckets upon an admin login
  • Amazon S3 must have permissions to create new buckets or bucket must be already created (AWS S3 Support doesn't work after migration into badgerdoc_storage #964)
  • Rename CI to use service/lib naming for workflows

@khyurri khyurri force-pushed the feature/add_blob_storage branch 3 times, most recently from bb89a52 to 584b2e2 Compare July 25, 2024 16:17
@khyurri khyurri changed the title wip: migrate all microservices from direct minio/boto3 usage to badgerdoc_storage feat: azure blob storage support Jul 25, 2024
khyurri and others added 24 commits July 29, 2024 13:31
@khyurri
feat: add azure blob storage support to badgerdoc_storage
2c45004
@djo753
test(annotation): cover row to dict function (#887)
66ba7aa 
Co-authored-by: Djordje Vukovic <djordje_vukovic@epam.com>
@djo753
test(annotation): cover main.py functions (#888)
19824f0 
Co-authored-by: Djordje Vukovic <djordje_vukovic@epam.com>
@Negoje
test(jobs): cover services.py functions (#892)
e607abe 
Co-authored-by: Filip Negojevic <filip_negojevic@epam.com>
@BogicevicMilan
test: Add parameterized test cases for validate_user_actions function (…
dcb72a8 
…#895)
@djo753
test(annotation): cover construct_annotated_doc (#896)
20dd6c8 
Co-authored-by: Djordje Vukovic <djordje_vukovic@epam.com>
@BogicevicMilan
test(annotation): cover tasks/services.py (#900)
f274af4
@Negoje
test(annotation): cover jobs.services functions (#904)
22a4d2c 
Co-authored-by: Filip Negojevic <filip_negojevic@epam.com>
@djo753
test(annotation): cover upload json and pages to minio (#897)
d27615f 
Co-authored-by: Djordje Vukovic <djordje_vukovic@epam.com>
@UkiStefanovic @cakeinsauce
test(annotation): cover distribution/main.py (#893)
d02df0e 
Co-authored-by: Uros Stefanovic <uros_stefanovic@epam.com>
Co-authored-by: Denis Rybakov <minefrs@gmail.com>
@djo753
test(annotation): cover update_pages_array and check_if_kafka_message…
b45d1db 
…_is_needed" (#898)

Co-authored-by: Djordje Vukovic <djordje_vukovic@epam.com>
@djo753
test(annotation): cover mark validated pages (#902)
b1380fb 
Co-authored-by: Djordje Vukovic <djordje_vukovic@epam.com>
@Negoje
test(annotation): cover jobs.services.py functions (#907)
8921fc4 
Co-authored-by: Filip Negojevic <filip_negojevic@epam.com>
@UkiStefanovic
test(annotation): cover distribute_tasks_extensively (#909)
0264d90 
Co-authored-by: Uros Stefanovic <uros_stefanovic@epam.com>
@djo753
test(annotation): cover load functions (#908)
15e0dee
@djo753
test(annotation): cover check null fields (#912)
7f8b995
@djo753
test(annotation): cover accumulate pages info (#913)
2d103f0
@khyurri
feat: remove convert microservice
988df96
@djo753
test(annotation): cover construct document links (#914)
49f5e5c
@UkiStefanovic
test(annotation): cover distribute_whole_files (#918)
cd4739d 
Co-authored-by: Uros Stefanovic <uros_stefanovic@epam.com>
@UkiStefanovic
test(annotation): cover create_tasks (#919)
f6e9e2b
@UkiStefanovic
test(annotation): cover distribute (#916)
96395cd
@Negoje @djo753
test(annotation): cover jobs.services.py functions (#917)
61046ab 
Co-authored-by: djo753 <145800147+djo753@users.noreply.github.com>
Co-authored-by: Filip Negojevic <filip_negojevic@epam.com>
@UkiStefanovic
test(annotation): cover module distribution (#920)
2af9e95
BogicevicMilan and others added 13 commits August 27, 2024 19:38
@BogicevicMilan
test(annotation): cover id and annotation tasks (#930)
daaec46
@BogicevicMilan
test(annotation): cover ids and links functions (#931)
4b241c3
@BogicevicMilan
test(annotation): cover common objs functions (#932)
70839ef
@BogicevicMilan
test(annotation): cover tasks without ids (#933)
705b55a
@khyurri
remove convert from docker-compose
f5121ba
@MilanBogicevic8
test(annotation): validation function (#949)
691f2cc
@MilanBogicevic8
test(annotation): task annotation pages (#951)
3e09815
@djo753
fix(tasks): add LRU class, remove lru-dict dependency (#940)
18c461a
@MilanBogicevic8
test(annotation): csv and lru functions (#953)
ea74f02
@UkiStefanovic
fix(annotation): remove kafka dependency (#943)
a15fc81
@Negoje
fix(annotation): rewrite jobs_communication sync to async http commun…
39ce681 
…ication (#941)

Co-authored-by: Filip Negojevic <filip_negojevic@epam.com>
@aazaliyaa
feat: Add datasets tab with new table (#944)
219c9e8
@khyurri @cakeinsauce
feat: add ci to badgerdoc_storage (#946)
bf182b0 
Co-authored-by: Denis Rybakov <minefrs@gmail.com>
@khyurri khyurri mentioned this pull request Sep 2, 2024
Closed
@khyurri khyurri mentioned this pull request Sep 6, 2024
Closed
@khyurri khyurri merged commit 8342a50 into main Sep 9, 2024
22 checks passed
@khyurri khyurri deleted the feature/add_blob_storage branch September 9, 2024 09:16
@khyurri khyurri mentioned this pull request Sep 11, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@cakeinsauce cakeinsauce cakeinsauce approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
9 participants
@khyurri @cakeinsauce @djo753 @Negoje @BogicevicMilan @UkiStefanovic @MilanBogicevic8 @aazaliyaa @ervandagadzhanyan