refactor(parquet): Use velox parquet reader in StatisticsTest and Metadatatest #11472
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
the
CLA Signed
✅ Deploy Preview for meta-velox canceled.
|
jkhaliqi
force-pushed
the
rmv_arrow_parquet_reader
branch
3 times, most recently
from
4af254d to
c97061c
Compare
majetideepak
reviewed
Nov 8, 2024
There was a problem hiding this comment.
@jkhaliqi can you move the file from arrow tests to here so that we can only see the Velox specific changes?
| return readerBase_->thriftFileMetaDatas(); | ||
| } | ||
|
|
||
| void WriteBufferToFile(const std::shared_ptr<arrow::Buffer>& buffer, const std::string& file_path) { |
There was a problem hiding this comment.
We don't need this here as this only used for testing.
There was a problem hiding this comment.
Ok, will change this methods location since it is only for testing before changing from draft PR, for now will use this
There was a problem hiding this comment.
deleted whole function
| @@ -74,6 +75,10 @@ class ReaderBase { | |||
| return *fileMetaData_; | |||
| } | |||
|
|
|||
| std::shared_ptr<thrift::FileMetaData> thriftFileMetaDatas() const { | |||
There was a problem hiding this comment.
We can use thriftFileMetaData() above. We don't need this.
There was a problem hiding this comment.
Updated to use above function, thank you!
jkhaliqi
force-pushed
the
rmv_arrow_parquet_reader
branch
4 times, most recently
from
d62718d to
495598e
Compare
majetideepak
reviewed
Nov 9, 2024
| } | ||
|
|
||
| void WriteBufferToFile(const std::shared_ptr<arrow::Buffer>& buffer, const std::string& file_path) { | ||
| auto source = std::make_shared<::arrow::io::BufferReader>(buffer); |
There was a problem hiding this comment.
You can simply write buffer()(returns a std::string_view) to a file using LocalWriteFile. See S3FileSystemTest.cpp for example.
majetideepak
reviewed
Nov 9, 2024
| ASSERT_NE(nullptr, file_reader->metadata()->key_value_metadata()); | ||
| auto read_kv_meta = file_reader->metadata()->key_value_metadata(); | ||
| // Write the buffer to a temp file | ||
| std::string file_path = "output_file.parquet"; |
jkhaliqi
force-pushed
the
rmv_arrow_parquet_reader
branch
6 times, most recently
from
8aa9724 to
85013aa
Compare
jkhaliqi
changed the title
jkhaliqi
force-pushed
the
rmv_arrow_parquet_reader
branch
10 times, most recently
from
287d618 to
d5f7423
Compare
jkhaliqi
changed the title
jkhaliqi
force-pushed
the
rmv_arrow_parquet_reader
branch
from
d5f7423 to
ece201e
Compare
jkhaliqi
force-pushed
the
rmv_arrow_parquet_reader
branch
from
ece201e to
abc8f1d
Compare
jkhaliqi
force-pushed
the
rmv_arrow_parquet_reader
branch
from
abc8f1d to
c862477
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.