s3DoesObjectExist Forbidden

[psimms-r7]

Jenkins and plugins versions report

Environment

Jenkins: 2.361.2
OS: Linux - 5.4.209-116.367.amzn2.x86_64
---
ace-editor:1.1
ansicolor:1.0.2
antisamy-markup-formatter:2.7
apache-httpcomponents-client-4-api:4.5.13-138.v4e7d9a_7b_a_e61
authentication-tokens:1.4
authorize-project:1.4.0
aws-credentials:191.vcb_f183ce58b_9
aws-java-sdk:1.12.287-357.vf82d85a_6eefd
aws-java-sdk-cloudformation:1.12.287-357.vf82d85a_6eefd
aws-java-sdk-codebuild:1.12.287-357.vf82d85a_6eefd
aws-java-sdk-ec2:1.12.287-357.vf82d85a_6eefd
aws-java-sdk-ecr:1.12.287-357.vf82d85a_6eefd
aws-java-sdk-ecs:1.12.287-357.vf82d85a_6eefd
aws-java-sdk-efs:1.12.287-357.vf82d85a_6eefd
aws-java-sdk-elasticbeanstalk:1.12.287-357.vf82d85a_6eefd
aws-java-sdk-iam:1.12.287-357.vf82d85a_6eefd
aws-java-sdk-logs:1.12.287-357.vf82d85a_6eefd
aws-java-sdk-minimal:1.12.287-357.vf82d85a_6eefd
aws-java-sdk-sns:1.12.287-357.vf82d85a_6eefd
aws-java-sdk-sqs:1.12.287-357.vf82d85a_6eefd
aws-java-sdk-ssm:1.12.287-357.vf82d85a_6eefd
badge:1.9.1
basic-branch-build-strategies:1.3.2
blueocean:1.25.8
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.25.8
blueocean-commons:1.25.8
blueocean-config:1.25.8
blueocean-core-js:1.25.8
blueocean-dashboard:1.25.8
blueocean-display-url:2.4.1
blueocean-events:1.25.8
blueocean-git-pipeline:1.25.8
blueocean-github-pipeline:1.25.8
blueocean-i18n:1.25.8
blueocean-jwt:1.25.8
blueocean-personalization:1.25.8
blueocean-pipeline-api-impl:1.25.8
blueocean-pipeline-editor:1.25.8
blueocean-pipeline-scm-api:1.25.8
blueocean-rest:1.25.8
blueocean-rest-impl:1.25.8
blueocean-web:1.25.8
bootstrap5-api:5.2.1-3
bouncycastle-api:2.26
branch-api:2.1046.v0ca_37783ecc5
build-with-parameters:1.6
buildtriggerbadge:251.vdf6ef853f3f5
caffeine-api:2.9.3-65.v6a_47d0f4d1fe
checks-api:1.7.5
cloudbees-bitbucket-branch-source:791.vb_eea_a_476405b
cloudbees-disk-usage-simple:178.v1a_4d2f6359a_8
cloudbees-folder:6.758.vfd75d09eea_a_1
command-launcher:90.v669d7ccb_7c31
commons-lang3-api:3.12.0-36.vd97de6465d5b_
commons-text-api:1.10.0-27.vb_fa_3896786a_7
config-file-provider:3.11.1
configuration-as-code:1559.v38a_b_2e3b_6b_b_7
configuration-as-code-groovy:1.1
credentials:1189.vf61b_a_5e2f62e
credentials-binding:523.vd859a_4b_122e6
datadog:5.0.0
display-url-api:2.3.6
docker-commons:1.21
durable-task:501.ve5d4fc08b0be
echarts-api:5.4.0-1
embeddable-build-status:255.va_d2370ee8fde
envinject:2.881.v37c62073ff97
envinject-api:1.199.v3ce31253ed13
extended-read-permission:3.2
external-monitor-job:203.v683c09d993b_9
favorite:2.4.1
font-awesome-api:6.2.0-3
git:4.12.1
git-changelog:3.23
git-client:3.12.1
github:1.35.0
github-api:1.303-400.v35c2d8258028
github-branch-source:1695.v88de84e9f6b_9
github-pr-comment-build:86.v23ae6d00ab99
groovy:453.vcdb_a_c5c99890
handlebars:3.0.8
handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953
htmlpublisher:1.31
instance-identity:116.vf8f487400980
ionicons-api:31.v4757b_6987003
jackson2-api:2.13.4-293.vee957901b_6fb
jakarta-activation-api:2.0.1-2
jakarta-mail-api:2.0.1-2
javadoc:226.v71211feb_e7e9
javax-activation-api:1.2.0-5
javax-mail-api:1.6.2-8
jaxb:2.3.6-2
jdk-tool:55.v1b_32b_6ca_f9ca
jenkins-design-language:1.25.8
jjwt-api:0.11.5-77.v646c772fddb_0
jnr-posix-api:3.1.15-2
job-dsl:1.81
jquery3-api:3.6.1-2
jsch:0.1.55.61.va_e9ee26616e7
junit:1153.v1c24f1a_d2553
kubernetes:3724.v0920c1e0ec69
kubernetes-client-api:5.12.2-193.v26a_6078f65a_9
kubernetes-credentials:0.9.0
kubernetes-credentials-provider:1.201.v11b_14c7a_0772
ldap:2.12
m2release:0.16.3
mailer:438.v02c7f0a_12fa_4
matrix-auth:3.1.5
matrix-project:785.v06b_7f47b_c631
maven-plugin:3.20
metrics:4.2.10-389.v93143621b_050
mina-sshd-api-common:2.9.1-44.v476733c11f82
mina-sshd-api-core:2.9.1-44.v476733c11f82
momentjs:1.1.1
monitoring:1.91.0
okhttp-api:4.9.3-108.v0feda04578cf
pipeline-aws:1.43
pipeline-build-step:2.18
pipeline-github:2.8-138.d766e30bb08b
pipeline-graph-analysis:195.v5812d95a_a_2f9
pipeline-groovy-lib:612.v84da_9c54906d
pipeline-input-step:451.vf1a_a_4f405289
pipeline-milestone-step:101.vd572fef9d926
pipeline-model-api:2.2114.v2654ca_721309
pipeline-model-definition:2.2114.v2654ca_721309
pipeline-model-extensions:2.2114.v2654ca_721309
pipeline-rest-api:2.26
pipeline-stage-step:296.v5f6908f017a_5
pipeline-stage-tags-metadata:2.2114.v2654ca_721309
pipeline-stage-view:2.26
pipeline-utility-steps:2.13.0
plain-credentials:139.ved2b_9cf7587b
plugin-util-api:2.18.0
popper2-api:2.11.6-2
prometheus:2.0.11
pubsub-light:1.17
rapid7-insightvm-container-assessment:1.0.21
remote-file:1.23
saferestart:0.3
saml:4.354.vdc8c005cda_34
scm-api:621.vda_a_b_055e58f7
script-security:1183.v774b_0b_0a_a_451
slack:625.va_eeb_b_168ffb_0
snakeyaml-api:1.32-86.ve3f030a_75631
sse-gateway:1.26
ssh-credentials:305.v8f4381501156
sshd:3.249.v2dc2ea_416e33
structs:324.va_f5d6774f3a_d
timestamper:1.20
token-macro:308.v4f2b_ed62b_b_16
trilead-api:2.72.v2a_3236754f73
variant:59.vf075fe829ccb
windows-slaves:1.8.1
workflow-aggregator:590.v6a_d052e5a_a_b_5
workflow-api:1200.v8005c684b_a_c6
workflow-basic-steps:994.vd57e3ca_46d24
workflow-cps:2802.v5ea_628154b_c2
workflow-durable-task-step:1206.v8a_d5f86e336b
workflow-job:1239.v71b_b_a_124a_725
workflow-multibranch:716.vc692a_e52371b_
workflow-scm-step:400.v6b_89a_1317c9a_
workflow-step-api:639.v6eca_cd8c04a_a_
workflow-support:838.va_3a_087b_4055b
xml-job-to-job-dsl:0.1.13

What Operating System are you using (both controller, and any agents involved in the problem)?

Environment is deployed into aws eks v1.23
Controller - jenkins/jenkins:2.361.2-jdk11
Agent - jenkins/inbound-agent:latest

Reproduction steps

We are now getting a forbidden error when using s3DoesObjectExist, this has been working for some time but recently randomly broke- I presume this just lists the file?

• I have given the bucket and iam role in use here full s3:* permissions
• Can list and push using aws cli in pipeline
• Can push using s3Upload function in pipeline

Example pipeline

pipeline {
    agent any

    stages {
        stage('Test') {
            steps {
                script {
                    sh "aws s3 ls s3://test-bucket-here/test"
                    s3ChartExist = s3DoesObjectExist(bucket:"test-bucket-here", path:"test")
                }
            }
        }
    }
}

Example output

Running on test-8-j4zkb-0qrpl-qqjmb in /home/jenkins/agent/workspace/test
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Test)
[Pipeline] script
[Pipeline] {
[Pipeline] sh
+ aws s3 ls s3://test-bucket-here/test
2022-10-20 11:43:05          0 test
[Pipeline] s3DoesObjectExist
Searching s3://test-bucket-here for object:'test'
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline
com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: ; S3 Extended Request ID: ; Proxy: null), S3 Extended Request ID: 
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403)
	at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1372)
	at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1346)
	at com.amazonaws.services.s3.AmazonS3Client.doesObjectExist(AmazonS3Client.java:1427)
	at de.taimos.pipeline.aws.S3DoesObjectExistStep$Execution.run(S3DoesObjectExistStep.java:115)
	at de.taimos.pipeline.aws.S3DoesObjectExistStep$Execution.run(S3DoesObjectExistStep.java:93)
	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Finished: FAILURE

Expected Results

s3DoesObjectExist to confirm if object exist and not be forbidden even though full permissions are given

Actual Results

com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden

Anything else?

No response

[paulmallott]

This appears to still be an issue.

One detail worth noting - I'm seeing the exception when using s3DoesObjectExist in a different AWS account. If the object is found, it returns true. If the object is not found, it throws the 403 exception.

However, if I use s3DoesObjectExist on the same object with an IAM user with the same permissions but from the same account, it returns false as expected.