Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

detect_nvmes feature doesn't work for AWS #407

Open
igorribeiroduarte opened this issue Sep 25, 2024 · 1 comment · May be fixed by #413
Open

detect_nvmes feature doesn't work for AWS #407

igorribeiroduarte opened this issue Sep 25, 2024 · 1 comment · May be fixed by #413

Comments

@igorribeiroduarte
Copy link
Collaborator

AWS instances have an extra NVME for the boot disk, so when you enable detect_nvmes on a cluster with AWS instances, the node role will try to also use this extra NVME, since by default we try to use all the NVMEs present in the instance.
We should change the role to only use NVMEs that are not already being used.

CC: @tarzanek

@vreniers
Copy link
Collaborator

vreniers commented Oct 7, 2024

The problem is a bit different than described. There is no "extra" NVMe on AWS instances.

When you use an AWS Instance that has local NVMe disks - or what they refer to as instance store volumes - there will also be a volume that shows up as NVMe but however isn't. This is the root/boot volume which is in fact an EBS volume, however when you run lsblk, it will show up as if it is an NVMe.

Some more information here: https://docs.aws.amazon.com/ebs/latest/userguide/nvme-ebs-volumes.html

This is even the fact when you add additional EBS volumes to the disk, they show up as NVMe.
Here's an example of an i4i.large where I set the root volume (EBS) to 30GB and also attached an EBS volume of 40GB.

nvme1n1      259:0    0    30G  0 disk 
├─nvme1n1p1  259:3    0  29.9G  0 part /
├─nvme1n1p14 259:4    0     4M  0 part 
└─nvme1n1p15 259:5    0   106M  0 part /boot/efi
nvme2n1      259:1    0 435.9G  0 disk 
nvme0n1      259:2    0    40G  0 disk 

There is a way to filter out which are really the NVMe, the shown serial using lsblk -o +SERIAL with EBS volumes shows the volume id, as explained here: https://docs.aws.amazon.com/ebs/latest/userguide/identify-nvme-ebs-device.html.

Either way, it would be a good start to indeed not use any "NVMe" that are not in use. I don't see an immediate way to filter it out using the nvme command, so perphaps lsblk is the way to go.

vreniers added a commit to vreniers/scylla-ansible-roles that referenced this issue Oct 11, 2024
NVMe disks should only be selected that are not in use, which should be the case when they are not partitioned.

Fixes scylladb#407
vreniers added a commit to vreniers/scylla-ansible-roles that referenced this issue Nov 5, 2024
NVMe disks should only be selected that are not in use, which should be the case when they are not mounted.
Fixes scylladb#407

Signed-off-by: Vincent Reniers <[email protected]>
vreniers added a commit to vreniers/scylla-ansible-roles that referenced this issue Dec 3, 2024
NVMe disks should only be selected that are not in use, which should be the case when they are not mounted.
Fixes scylladb#407

Signed-off-by: Vincent Reniers <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants