Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update of start-up script to download tes_vm_perf.tgz and execute bootstrapper #622

Draft
wants to merge 48 commits into
base: main
Choose a base branch
from
Draft

Update of start-up script to download tes_vm_perf.tgz and execute bootstrapper #622

wants to merge 48 commits into from

Conversation

jlester-msft
Copy link
Contributor

@jlester-msft jlester-msft commented Feb 27, 2024 •
edited by BMurri
Loading

No description provided.

jlester-msft and others added 28 commits February 27, 2024 15:30
@jlester-msft
Update of start-up script to download tes_vm_perf.tgz and execute boo…
a2e5c16 
…tstrapper
@jlester-msft
Side-step build tests
1a3da64
@jlester-msft
Add dummy function for the correct call
c663328
@jlester-msft
add catch for GetInternalTesBlobUrlAsync
1712d1c
@jlester-msft
chmod typo fixed
3618aea
@MattMcL4475
Add script to be uploaded every time
70eff6a
@MattMcL4475
remove try/catch
2d20d3d
@jlester-msft
extract explictly to AZ_BATCH_TASK_DIR
bd0c9eb
@jlester-msft
disable extraction and running integration test
fa98c05
@jlester-msft
Add absolute paths for tar
ffdae36
@jlester-msft
Wrap batch_script execution in a function, call it with screen, and u…
baaf634 
…pload the log to the storage account task dir
@jlester-msft
Updating bash script
3964c62
@jlester-msft
Update scripts and use python to parse nodetaskurl
901e207
@jlester-msft
Update python script, msi auth for azcopy, add task complete, and add…
d1a294c 
… exit code capture
@jlester-msft
missing semicolon
e5c9cb2
@jlester-msft
remove windows line endings
bc0e7db
@jlester-msft
batchNodeScript is ending up with Windows Style line endings which is…
c4733d3 
… confusing bash. remove extra spaces to avoid

$'\r': command not found errors
@jlester-msft
export env and load it inside screen. Also run outside screen once
432c8d4
@jlester-msft
fix missing quote
5316a4b
@jlester-msft
convert script to @Strings
6750b7d
@jlester-msft
Add shebang disable direct script call
e3e0689
@jlester-msft
Cleanup of bash_script construction, build pipeline test
459b90f
@jlester-msft
remove unused functions
0331bdd
@jlester-msft
Add additional debugging capture of host system on failure, change st…
9049421 
…art task
@jlester-msft
Add debugging tools
4c224a7
@jlester-msft
Always collect debug info
510268c
@jlester-msft
Update naming of perf archive to tes_vm_monitor.tar.gz
72a81fd 
Upload debug information using single azcopy on error only
add notes for making setup of vm_monitor work in the start task
@jlester-msft
Merge branch 'main' into jlester/vm_node_perf
38a986f
MattMcL4475
MattMcL4475 reviewed Mar 25, 2024
View reviewed changes
src/vm_monitor/vm_monitor_scripts/collect_azure_vm_perf.sh
@@ -0,0 +1,13 @@
#!/bin/bash
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

JL to delete

@MattMcL4475
Copy link
Collaborator

MattMcL4475 commented Mar 25, 2024 •
edited by BMurri
Loading

TODO:

  • 1. Add configuration setting; something like AdvancedVmPerformanceMonitoringEnabled; default false to the "BatchNodes" area
  • 2. Move the download to the start task
  • 3. Merge origin/main into this branch

@BMurri BMurri changed the title Update of start-up script to download tes_vm_perf.tgz and execute boo… Update of start-up script to download tes_vm_perf.tgz and execute bootstrapper Mar 25, 2024
@BMurri BMurri added the Needs Issue PR needs at least one associated issue label Mar 26, 2024
@BMurri
Copy link
Collaborator

BMurri commented Apr 15, 2024

I'd prefer if we could build the binary download packages in the build rather than check those binaries into git

@jlester-msft
Add url validation to azure_append_blob
59f3a31 
Update dockerfile caching
Update telegraf configs with better timestamp behavior and collection cleanup
@jlester-msft
Copy link
Contributor Author

I'd prefer if we could build the binary download packages in the build rather than check those binaries into git

Agreed this would need some integration into the build pipeline to generate the binary package.

Right now src/vm_monitor/build_tes_deployment_archive.sh contains an example script that will do the building of the telegraf binary + build a .tar.gz of everything. The basic steps are:

  1. Build a custom stripped down version of telegraf
    1. The local build process should probably be skipped in favor of the docker style build in src/vm_monitor/build_telegraf_remote.sh. This is setup to build on a remote VM but doesn't need to. It is basically dockerfile build + run the docker to copy out the telegraf binary
  2. Make a flattened .tar.gz file with the binary + the scripts needed
  3. Put that binary somewhere consumable

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@MattMcL4475 MattMcL4475 MattMcL4475 left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
Needs Issue PR needs at least one associated issue
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@jlester-msft @MattMcL4475 @BMurri