Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Register write filesink for abfs connector #11973

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

zhli1142015
Copy link
Contributor

No description provided.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Dec 27, 2024
Copy link

netlify bot commented Dec 27, 2024

Deploy Preview for meta-velox canceled.

Name Link
🔨 Latest commit 6e475e2
🔍 Latest deploy log https://app.netlify.com/sites/meta-velox/deploys/676e3de4c3e8780008aed559

@zhli1142015
Copy link
Contributor Author

When creating an instance of AbfsWriteFile, we need to check if the file exists. Therefore, even when testing only with dwio::common::FileSink::create, we must replace the real client with a mock client as well.

cc @majetideepak

@zhli1142015
Copy link
Contributor Author

UT failure is not related to this PR.

290/398 Test #380: velox_hdfs_file_test .................................................***Failed   65.24 sec
Running main() from /__w/velox/velox/_build/release/_deps/gtest-src/googletest/src/gtest_main.cc
[==========] Running 21 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 20 tests from HdfsFileSystemTest
WARNING: HADOOP_PREFIX has been replaced by HADOOP_HOME. Using value of HADOOP_PREFIX.
WARNING: HADOOP_PREFIX has been replaced by HADOOP_HOME. Using value of HADOOP_PREFIX.
WARNING: Logging before InitGoogleLogging() is written to STDERR
E20241227 07:04:46.299198 14359 Exceptions.h:66] Line: /__w/velox/velox/velox/connectors/hive/storage_adapters/hdfs/tests/HdfsMiniCluster.cpp:92, Function:addFile, Expression:  Failed to add file to hdfs, exit code: 383, Source: RUNTIME, ErrorCode: INVALID_STATE
unknown file: Failure
C++ exception with description "Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Failed to add file to hdfs, exit code: 383
Retriable: False
Function: addFile
File: /__w/velox/velox/velox/connectors/hive/storage_adapters/hdfs/tests/HdfsMiniCluster.cpp
Line: 92
Stack trace:
# 0  _ZN8facebook5velox7process10StackTraceC1Ei
# 1  _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2  _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEvRKNS1_18VeloxCheckFailArgsET0_
# 3  _ZN8facebook5velox11filesystems4test15HdfsMiniCluster7addFileENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES9_
# 4  _ZN18HdfsFileSystemTest14SetUpTestSuiteEv
# 5  _ZN7testing8internal35HandleExceptionsInMethodIfSupportedINS_9TestSuiteEvEET0_PT_MS4_FS3_vEPKc
# 6  _ZN7testing9TestSuite3RunEv
# 7  _ZN7testing8internal12UnitTestImpl11RunAllTestsEv
# 8  _ZN7testing8UnitTest3RunEv
# 9  main
# 10 __libc_start_call_main
# 11 __libc_start_main
# 12 _start
" thrown in SetUpTestSuite().

Copy link
Collaborator

@majetideepak majetideepak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zhli1142015 Thanks for adding the tests. A couple of comments.

@@ -23,6 +23,43 @@

namespace facebook::velox::filesystems {

std::unique_ptr<AzureDataLakeFileClient> AbfsConfig::fakeWriteClient_;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's use test instead of fake here and elsewhere.

@@ -92,6 +93,12 @@ class AbfsConfig {
return authorityHost_;
}

/// Test only.
static void registerFakeWriteFileClient(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's say createTestWriteFileClient?

/// Test only.
static void registerFakeWriteFileClient(
std::unique_ptr<AzureDataLakeFileClient> fakeClient) {
fakeWriteClient_ = std::move(fakeClient);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For safety and simplicity, let's check and initialize the singleton testWriteClient_ here.

@majetideepak majetideepak requested a review from JkSelf December 27, 2024 22:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants