Bug 1922986 - Create script that exports BMO data as JSON suitable for import into a BigQuery instance in GCP #2370
Conversation
…r import into a BigQuery instance in GCP
…g ./extensions/BMO/bin/export_bmo_etl.pl
* upstream/master: Bumped version to 20241211.1
| # to make sure other entries with this date are not already present. | ||
| check_for_duplicates(); | ||
|
|
||
| ### Bugs |
There was a problem hiding this comment.
The code related to each individual table should be moved to a separate function, like extract_bugs, instead of being one large script separated by comments. This goes for attachments, flags, keywords etc.
Nit: also, looking at each of these sections, it seems there is some repeated structure amongst them. Each one preps a statement, runs a while loop over the total number of rows, fetches a row, checks the cache, then extracts the data and submits. You could eliminate that repetition.
| $table_name = 'bug_mentors'; | ||
| $rows | ||
| = $dbh->selectall_arrayref( | ||
| 'SELECT bug_id, user_id FROM bug_mentors ORDER BY bug_id', | ||
| {Slice => {}}); | ||
|
|
||
| $total = scalar @{$rows}; | ||
| $count = 0; | ||
|
|
||
| print "Processing $total $table_name.\n" if $verbose; | ||
|
|
||
| @results = (); | ||
| foreach my $row (@{$rows}) { | ||
| my $data = {bug_id => $row->{bug_id}, user_id => $row->{user_id}}; | ||
|
|
||
| push @results, $data; | ||
|
|
||
| $count++; | ||
|
|
||
| # Send the rows to the server if we have a specific sized block' | ||
| # or we are at the last row | ||
| if (scalar @results == API_BLOCK_COUNT || $total == $count) { | ||
| send_data($table_name, \@results, $count); | ||
| @results = (); | ||
| } | ||
| } | ||
|
|
There was a problem hiding this comment.
These sections should also be moved to separate functions.
There is room for abstracting away the general structure here as well, lots of shared code between "mentors"/"see also"/etc.
extensions/BMO/template/en/default/hook/admin/params/editparams-current_panel.html.tmpl
Show resolved
Hide resolved
|
I need to make some other non-related changes anyway as I realized i left out two additional requirements. 1) I need to skip sending bug data related to the 'Legal' product and 2) I need to filter the summaries, etc. for bugs that are private. |
Sorry this so late. Lots of moving pieces had to be coordinated with webservices-infra, etc. just so I could test and then small changes one after another to get to this point.
Let me know if you have any questions as it is a lot.