Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DM-41032: Add per-dataset-type special transfer handling #24

Merged
merged 76 commits into from
Mar 6, 2024
Merged

DM-41032: Add per-dataset-type special transfer handling #24

merged 76 commits into from
Mar 6, 2024

Conversation

beckynevin
Copy link
Contributor

New features:

  • accepts a list of datasettypes and collections
  • tests the above
  • all other tests still pass when datasettypes is a string
  • splits the input list of datasettypes into two categories: those with and without exposures
  • Those with exposures (ie raw type) must also register URI and then runs through Tim's modified version of transfer_from which prepares for transfer
  • Those with exposures are also run through transfer_dimension_records_from
  • Finally, ingest
  • All of the before code works for the no exposure type, although we don't currently have any of this datatype to test (that'll happen in 41033, which we'll rebase off of this branch once it's merged to main)

jsv1206 and others added 30 commits September 28, 2023 20:20
@jsv1206
testing query for different dtypes
139359f
@jsv1206
tranfer datatypes without exposure or visit dimension
5a09b06
@beckynevin
making is it there general
8b21c66
@beckynevin
changing to KTs deepcoadds
ed2dae6
@beckynevin
adding a test for new datatypes, probably we will walk back on this b…
26b8af8 
…ecause we want the program to accept a list of datatypes and collections instead of testing for them individually
@beckynevin
will not run workflow
954b4f3
@beckynevin
should be able to deal with a list
8b9ef0f
@beckynevin
starting to modify tests but might need to git add some .fits files o…
abf9f4f 
…f another type
@jsv1206
tesing ingesting raw datatype into scratch butler
8200210
@beckynevin
adding if statements to run transfer if datalist exists for exposure …
c9d7723 
…and/or ingest type
@beckynevin
runs through with same key error
dc066cd
@beckynevin
modifying dimension records
b400c2a
@beckynevin
ingest works with no errors
2d3a4de
@jsv1206
prep_for_ingest function which registers datasetType and collection a…
1aa0687 
…nd transfers dimensions to target butler
@jsv1206
changed registry to source_registry in prep_for_ingest function
fa30cce
@jsv1206
preparing for ingest in a separate function that can be called in the…
7d555d7 
… main script
@beckynevin
changed variables to be datalist_no_exposure, added prep for ingest a…
36b0e4e 
…nd filedatalist for raws
@beckynevin
adding dest uri prefix as a command line input
3300e38
@beckynevin
changing list of datasettypes to a string input
876f9ba
@beckynevin
adding desturi as an option, adding assertion if this is not defined
ae41ace
@beckynevin
trying to troubleshoot locally
acce1a9
@beckynevin
trying to run testing on this branch
b7abc77
@beckynevin
adding desturiprefix as an argument to is_it_there, still getting the…
f36d4aa 
… registry does not exist error
@jsv1206
error reading in list of datasettypes as argparse argument
2e78bd9
@jsv1206
trying different ways to input list of strongs to subprocess
f977576
@beckynevin
its kind of working, within move embargo args its [raw,calexp] so alm…
98cb035 
…ost what we want ([raw,calexp])
@beckynevin
okay datasettypelist is proper format in move_embargo_args
edf7642
@jsv1206
modified prep_transfer collection according to butler transfer_from
3a9e42c
@beckynevin
adding collections as a list option
69ceaf9
@beckynevin
pushing to trigger workflow
96a84c6
beckynevin and others added 21 commits January 19, 2024 20:40
@beckynevin
trying to install just obs_lsst
8010010
@beckynevin
daf butler now a second eups install and setup command
4db0b80
@beckynevin
adding setup obs_lsst
848e3c8
@jsv1206
setup in run tests
fea7bd9
@jsv1206
setup in run tests
f7d41cd
@beckynevin
changing mapping back to full this could be an error with version mis…
aba8131 
…match of lsst
@beckynevin
changing from v26.0.0 to w_2023_50 and backo .mapping
b3df118
@beckynevin
correcting flake8 errors in move_embargo_args
4b78d36
@beckynevin
correcting flake8 errors on test_move_embargo
ee55f97
@beckynevin
adding cache workflow
a3f0418
@beckynevin
indenting restore keys for proper yml syntax on cache file
bcd572b
@beckynevin
rm-ing the non-cached workflow
52aad17
@beckynevin
changing name of workflow
b8e5124
@beckynevin
removing python dependencies
7e86666
@beckynevin
workflow should now not run on this branch
de1bcc0
@beckynevin
now have an additional test that makes sure you're going through the …
7abaaae 
…loop
@beckynevin
saved the test that only passes if it fails
66ab6e7
@beckynevin
correcting flake8 errors in test suite
aabfe5f
@beckynevin
modified test to have a try and exception if indexerror is out of lis…
90e7d80 
…t ie if the list is empty
@beckynevin
all tests passing locally
78354e9
@beckynevin
flake8 on new move embargo args mod
492b147
ktlim
ktlim reviewed Feb 27, 2024
View reviewed changes
.github/workflows/test_and_build_cache.yml Outdated Show resolved Hide resolved
.github/workflows/test_and_build_cache.yml Outdated Show resolved Hide resolved
.github/workflows/requirements.txt Outdated Show resolved Hide resolved
.github/workflows/test_and_build_cache.yml Outdated Show resolved Hide resolved
src/move_embargo_args.py Show resolved Hide resolved
src/move_embargo_args.py Show resolved Hide resolved
src/move_embargo_args.py Show resolved Hide resolved
src/move_embargo_args.py Outdated Show resolved Hide resolved
src/move_embargo_args.py Show resolved Hide resolved
src/move_embargo_args.py Show resolved Hide resolved
ktlim
ktlim reviewed Feb 27, 2024
View reviewed changes
tests/test_move_embargo_args.py Outdated Show resolved Hide resolved
tests/test_move_embargo_args.py Outdated Show resolved Hide resolved
tests/test_move_embargo_args.py Outdated Show resolved Hide resolved
tests/test_move_embargo_args.py Outdated Show resolved Hide resolved
@ktlim ktlim changed the title Tickets/dm 41032 DM-41032: Add per-dataset-type special transfer handling Feb 27, 2024
@beckynevin
addressing PR notes from KT for 41032; including lsst logging, fixing…
41429f2 
… some tests to use xfail, also touched .fits files to make them empty

git rm'ing requirements.txt

workflow uses environmental variable for EUPS version for both caching and installing dependencies related to eups

running isort and black on move

removing commented out code from move

removing commented out code from test also isort and black

removing all if statement for namespace.log

removing all print statements in favor lof logging

testing yaml syntax

syntax needs to be a string in yaml?

syntax back to pipes?

fixed indent in yaml

expunging prints and implementing logging and testing it works for info

going back to lsst clilog method

black on move

should run workflow

no more just import lsst

Union --> |, grabbing another lsst import

changing workflow tests to pytest

testing now expected failing

adding mechanism so that move is a store_true argument and so log is a string (INFO)

isort and black for test and move

resolving issue with dest uri and combining prune into one list

resolving dest uri

decided to add back f statements to asserts

making raw blank

raw 61 blank now

touching all .fits
@beckynevin
all tests run
b693f21
@beckynevin beckynevin merged commit a56e48f into main Mar 6, 2024
4 checks passed
@beckynevin beckynevin deleted the tickets/DM-41032 branch March 6, 2024 15:31
@beckynevin beckynevin restored the tickets/DM-41032 branch March 22, 2024 16:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ktlim ktlim ktlim left review comments

@timj timj timj left review comments

Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@beckynevin @timj @ktlim @jsv1206