Merge pull request #37 from pravega/SDPT-587

Added Boomi public libs

README.md

@@ -2,39 +2,19 @@
 A Pravega connector for the [Boomi Atomsphere](https://boomi.com/platform/integration/applications/)
 
 # Building
-_Note: once the connector is published, it will be available to everyone in the Boomi Platform, and you can skip to the [Custom Connector Library](#custom-connector-library) section._
+_Note: once the connector is published, it will be available to everyone in the Boomi Platform, and you can use it without building or creating a custom connector._
 
 ## Pre-requisites
 * Java JDK
-* Boomi Connector SDK
-* Maven
-
-### Install the Boomi Connector SDK
-1. Download and unzip the Boomi Connector SDK (you must have a Boomi account)
-1. In the unzipped folder, run the `install-boomi-sdk-local.sh` script (from this repository)
-
-This will use Maven to install the Connector libraries into your local Maven repository.  If you are on Windows, you can run the following commands directly:  
-```
-mvn install:install-file -Dfile=boomi-util-2.3.8.jar -DgroupId=com.boomi -DartifactId=boomi-util -Dversion=2.3.8 -Dpackaging=jar
-mvn install:install-file -Dfile=common-sdk-1.1.7.jar -DgroupId=com.boomi -DartifactId=common-sdk -Dversion=1.1.7 -Dpackaging=jar
-mvn install:install-file -Dfile=connector-sdk-api-2.3.0.jar -DgroupId=com.boomi -DartifactId=connector-sdk-api -Dversion=2.3.0 -Dpackaging=jar
-mvn install:install-file -Dfile=connector-sdk-util-2.3.0.jar -DgroupId=com.boomi -DartifactId=connector-sdk-util -Dversion=2.3.0 -Dpackaging=jar
-mvn install:install-file -Dfile=connector-sdk-test-util-2.3.0.jar -DgroupId=com.boomi -DartifactId=connector-sdk-test-util -Dversion=2.3.0 -Dpackaging=jar
-```
-(use the appropriate version number)
 
 ## Build artifacts
 ```
 ./gradlew distZip
 ```
-This will build a connector package which you can upload to your Boomi account. Note that this jar only includes the connector classes and no dependencies. This is because there is a 20MB size limitation on connector package uploads in the Boomi platform site and the connector dependencies total more than that (about 13MB). You will need to upload and create a custom library for the connector in your Boomi account before it will function properly.
-```
-./gradlew dependencyJar
-```
-This will build (only for version 1.0.0) a fat jar with all of the dependencies required for the connector to work. Upload this as an account library and create a custom library for the Pravega connector and deploy it to any environment that will run the connector.
+This will build a connector package which you can upload to your Boomi account (see the [Create a Custom Connector](#custom-connector-library) section below).
 
 # Create a Custom Connector
-_Note: once the connector is published, it will be available to everyone in the Boomi Platform, and you can skip to the [Custom Connector Library](#custom-connector-library) section._
+_Note: once the connector is published, it will be available to everyone in the Boomi Platform, and and you can use it without building or creating a custom connector._
 
 If you have just built the Pravega connector above, you must install it in your Boomi account as a custom connector type.  More details about this process can be found [here](https://help.boomi.com/bundle/connectors/page/c-atm-Connector_versioning_and_releasing_4ef53f03-4e3d-4637-9046-aa5f8b9506ba.html).  The basic steps are:
 
@@ -52,47 +32,7 @@ If you have just built the Pravega connector above, you must install it in your
     * Give the connector a classification (i.e. "prod", "test", etc.)
     * Click `OK`
 
-Now you should have a Pravega connector for your account that you can insert into any process.  However, before you can test or execute any of these processes, you will need to deploy the dependency libraries first (see next section).  
-
-# Custom Connector Library
-_Note: this step is necessary because the dependency libraries of this connector (when compressed in a fat jar) are larger than 20MiB, which is a current limit within the Boomi platform. In future, this limit will be increased and this step will no longer be necessary._
-
-Before you can deploy any process that uses the Pravega connector, you must first create and deploy a custom library that contains its dependencies. This is accomplished in 3 steps (the process is described in more detail [here](https://help.boomi.com/bundle/integration/page/c-atm-Working_with_custom_libraries_96f10864-334e-4eba-ac3f-f52b4e65fdb2.html)):
-
-1. Upload dependency jar
-    * [download](#dependency-jar-download) the appropriate dependency jar that matches the version of the connector you want to deploy
-    * Go to your Boomi account setup and under `Development Resources` on the left, click on `Account Libraries`
-    * Click on "Upload a File" and select the dependency jar you downloaded in the above step
-1. Create custom connector library
-    * In your Boomi account, click on `Build`, then `+ New` and select `Custom Library`
-    * Call your custom library something meaningful, like "Pravega Connector Dependencies v2" (use appropriate version)
-    * Click `Create`
-    * Select the `Connector` library type, and choose the Pravega connector you created above
-    * Select the dependency jar that you uploaded
-    * Click `Save and Close`
-1. Create packaged components
-    * In your Boomi account, click on `Deploy`, then below that, select `Packaged Components` and then `Create Packaged Components`
-    * Click on the custom library you created from the left panel, then click on `Add Details`
-    * Give a version number(optional) and then click on `Create Packaged Components`
-1. Deploy the packaged components
-    * In your Boomi account, click on `Deploy`, then below that, select `Deployments`
-    * Click on `Deploy Packaged Componenet`
-    * Select any environments on the right that will execute the Pravega connector
-    * Click on the `Select Version`
-    * Click on the packaged components you created from the left panel, then click on `Review`
-    * Click `Deploy` 
-
-At this point, you should be able to build and execute processes using the Pravega connector, pointing at an accessible Pravega storage cluster.
-
-# Dependency Jar Download
-
-If you require a custom connector library to deploy the Pravega connector (read above), then you can choose the appropriate library to download here.
-
-|Github Release Version|Boomi Platform Version|Download Link|Checksums|
-|---:|---:|---|---|
-|1.0.0|2|[boomi-pravega-connector-1.0.0-dependencies.jar](https://132173853047869709.public.ecstestdrive.com/pravega-boomi/boomi-pravega-connector-1.0.0-dependencies.jar)|MD5: c604f50683d8df50c38f480bc8ae0fbd<br>SHA1: 845aa1a9c792553d42a01b59587a45f7d279febe<br>SHA256: 35c4e5e8b4ad0ba66d7b2d52218cd77ebd2208952ab22a69a1245fcc5c7e895b|
-|2.0.0|2| No dependency jar needed.||
-|2.1.0|2|[netty-tcnative-boringssl-static-2.0.17.Final.jar](https://repo1.maven.org/maven2/io/netty/netty-tcnative-boringssl-static/2.0.17.Final/netty-tcnative-boringssl-static-2.0.17.Final.jar)|MD5: 60f88be0fa76bc2d1b89f71725910441<br>SHA1: b1e5acbde8c444c656131238ac6ab9e73f694300<br>SHA256: a71bf392a955bbde8665fc5d44abff8e3fef3dcee74c2821aa5690d7e65bc2be|
+Now you should have a Pravega connector for your account that you can insert into any process.
 
 # Using the Connector
 

build.gradle

@@ -6,11 +6,15 @@ plugins {
 }
 
 repositories {
-    mavenLocal()
+
     mavenCentral()
     maven {
         url "https://oss.jfrog.org/jfrog-dependencies"
     }
+    maven {
+        name = "Connector SDK Repository"
+        url = "https://boomisdk.s3.amazonaws.com/releases"
+    }
 }
 
 group = "io.pravega"
@@ -21,20 +25,18 @@ sourceCompatibility = 1.8
 targetCompatibility = 1.8
 
 dependencies {
-    compile("io.pravega:pravega-client:${pravegaVersion}") {
-        exclude group: "io.netty", module: "netty-tcnative-boringssl-static"
-    }
+    compile("io.pravega:pravega-client:${pravegaVersion}")
     compile "io.pravega:pravega-keycloak-client:${pravegaKeycloakVersion}"
     compile "com.jayway.jsonpath:json-path:${jsonPathVersion}"
-    compile "com.boomi:connector-sdk-util:${boomiConnectorVersion}"
-    compileOnly "com.boomi:common-sdk:${boomiCommonSdkVersion}"
-    compileOnly "com.boomi:connector-sdk-api:${boomiConnectorVersion}"
+    compile "com.boomi.connsdk:connector-sdk-util:${boomiConnectorVersion}"
+    compileOnly "com.boomi.commonsdk:common-sdk:${boomiCommonSdkVersion}"
+    compileOnly "com.boomi.connsdk:connector-sdk-api:${boomiConnectorVersion}"
     testImplementation "org.junit.jupiter:junit-jupiter:5.5.2"
     testCompile "io.pravega:pravega-standalone:${pravegaVersion}"
-    compile "com.boomi:boomi-util:${boomiUtilVersion}"
-    testCompile "com.boomi:common-sdk:${boomiCommonSdkVersion}"
-    testCompile "com.boomi:connector-sdk-api:${boomiConnectorVersion}"
-    testCompile "com.boomi:connector-sdk-test-util:${boomiConnectorVersion}"
+    compile "com.boomi.util:boomi-util:${boomiUtilVersion}"
+    testCompile "com.boomi.commonsdk:common-sdk:${boomiCommonSdkVersion}"
+    testCompile "com.boomi.connsdk:connector-sdk-api:${boomiConnectorVersion}"
+    testCompile "com.boomi.connsdk:connector-sdk-test-util:${boomiConnectorVersion}"
     testCompile "org.slf4j:slf4j-api:1.7.9"
     testCompile "ch.qos.logback:logback-core:1.2.3"
     testRuntime "ch.qos.logback:logback-classic:1.2.3"

doc/Pravega Connector User's Guide.md

@@ -113,7 +113,7 @@ Read Operation Options:
 - **Read Timeout** : The maximum amount of time the reader will wait for the next event from Pravega (in milliseconds).
 - **Max Events per Execution** : For each execution of the operation, the maximum number of events the reader will read before moving to the next component in the process. For example, if this is set to 100000 events, the operation will read a maximum of 100000 events. If it hits the tail of the stream (no more events), or this limit (100000 events), it will exit execution and the collection of events will be sent to the next component in the process flow. If this is set to 0, the operation will continue reading from the stream until it hits the tail (there are no more events). Note that if there are a lot of events, this may overload the Boomi document cache.
 - **Max Read Time per Execution** : For each execution of the operation, the maximum number of seconds the reader will read and collect events from the stream before moving to the next component in the process. For example, if this is set to 30 seconds, the operation will spend a maximum of 30 seconds reading events from the stream. If it hits the tail of the stream (no more events), or this time (30 seconds), it will exit execution and the collection of events will be sent to the next component in the process flow. If this is set to 0, the operation will continue reading from the stream until it hits the tail (there are no more events). Note that if there are a lot of events, this may overload the Boomi document cache.
-- **Initial Reader Group Position** : The initial position from which to start reading from the stream. A value of "Head" means the first time the listener runs, it will read from the head (beginning) of the stream. NOTE: this will read the entire stream from start to finish and may cause a very high load while doing so. A value of "Tail" means the first time the listener runs, it will start reading from the tail (end) of the stream. Use this option to start reading real-time events, but be aware that all events that were written before the first process execution will be ignored.
+- **Initial Reader Group Position** : The initial position from which to start reading from the stream. A value of "Head" means the first time the reader runs, it will read from the head (beginning) of the stream. NOTE: this will read the entire stream from start to finish and may cause a very high load while doing so. A value of "Tail" means the first time the reader runs, it will start reading from the tail (end) of the stream. Use this option to start reading real-time events, but be aware that all events that were written before the first process execution will be ignored.
 
 ## Listen
 

gradle.properties

@@ -1,8 +1,8 @@
-releaseVersion=2.1.0
+releaseVersion=2.1.1
 
-pravegaVersion=0.7.0
-pravegaKeycloakVersion=0.7.0
+pravegaVersion=0.8.0
+pravegaKeycloakVersion=0.8.0
 jsonPathVersion=2.4.0
 boomiUtilVersion=2.3.8
-boomiCommonSdkVersion=1.1.7
-boomiConnectorVersion=2.3.0
+boomiCommonSdkVersion=1.1.9
+boomiConnectorVersion=2.8.0

src/main/java/io/pravega/connector/boomi/PravegaBrowser.java

@@ -13,13 +13,8 @@
 import com.boomi.connector.api.*;
 import com.boomi.connector.util.BaseBrowser;
 
-import java.util.WeakHashMap;
-
 public class PravegaBrowser extends BaseBrowser implements ConnectionTester {
 
-    //store keycloak.json contents file as key and file path as value
-    private WeakHashMap<String, String> map = new WeakHashMap<>();
-
     public PravegaBrowser(BrowseContext browseContext) {
         super(browseContext);
     }
@@ -37,7 +32,7 @@ public ObjectTypes getObjectTypes() {
     @Override
     public void testConnection() {
         try {
-            PravegaUtil.testConnection(getContext(), PravegaUtil.checkAndSetCredentials(getContext(), map));
+            PravegaUtil.testConnection(getContext(), PravegaUtil.getKeycloakCredentialsString(getContext()));
         } catch (Throwable t) {
             throw new ConnectorException("Could not initialize connection to Pravega", t);
         }

src/main/java/io/pravega/connector/boomi/PravegaConfig.java

@@ -25,15 +25,15 @@ public class PravegaConfig {
     private AuthenticationType authMethod;
     private String userName;
     private String password;
-    private String keycloakJSONPath;
+    private String keycloakJSONString;
     private boolean createScope;
     private long interval;
     private TimeUnit unit;
 
     public PravegaConfig() {
     }
 
-    public PravegaConfig(ConnectorContext context, String keycloakJsonPath) {
+    public PravegaConfig(ConnectorContext context, String keycloakJsonString) {
         Map<String, Object> props = context.getConnectionProperties();
 
         // URI, scope, and stream should always be set
@@ -56,7 +56,7 @@ public PravegaConfig(ConnectorContext context, String keycloakJsonPath) {
             setAuth(AuthenticationType.valueOf(auth));
         setUserName((String) props.get(Constants.USER_NAME_PROPERTY));
         setPassword((String) props.get(Constants.PASSWORD_PROPERTY));
-        setKeycloakJSONPath(keycloakJsonPath);
+        setKeycloakJSONString(keycloakJsonString);
         setInterval((long) props.get(Constants.INTERVAL));
         setUnit((String) props.get(Constants.TIME_UNIT));
     }
@@ -116,12 +116,12 @@ public void setPassword(String password) {
         this.password = password;
     }
 
-    public String getKeycloakJSONPath() {
-        return keycloakJSONPath;
+    public String getKeycloakJSONString() {
+        return keycloakJSONString;
     }
 
-    public void setKeycloakJSONPath(String keycloakJSONPath) {
-        this.keycloakJSONPath = keycloakJSONPath;
+    public void setKeycloakJSONString(String keycloakJSONString) {
+        this.keycloakJSONString = keycloakJSONString;
     }
 
     public boolean isCreateScope() {

src/main/java/io/pravega/connector/boomi/PravegaConnector.java

@@ -14,31 +14,26 @@
 import com.boomi.connector.api.listen.ListenOperation;
 import com.boomi.connector.util.listen.BaseListenConnector;
 
-import java.util.WeakHashMap;
-
 public class PravegaConnector extends BaseListenConnector {
 
-    //store keycloak.json contents file as key and file path as value
-    private WeakHashMap<String, String> map = new WeakHashMap<>();
-
     @Override
     protected Operation createQueryOperation(OperationContext context) {
-        return new PravegaReadOperation(context, PravegaUtil.checkAndSetCredentials(context, map));
+        return new PravegaReadOperation(context, PravegaUtil.getKeycloakCredentialsString(context));
     }
 
     @Override
     protected Operation createCreateOperation(OperationContext context) {
-        return new PravegaWriteOperation(context, PravegaUtil.checkAndSetCredentials(context, map));
+        return new PravegaWriteOperation(context, PravegaUtil.getKeycloakCredentialsString(context));
     }
 
     @Override
     public ListenOperation<PravegaPollingManager> createListenOperation(OperationContext context) {
-        return new PravegaPollingOperation(new PravegaPollingOperationConnection(context, PravegaUtil.checkAndSetCredentials(context, map)));
+        return new PravegaPollingOperation(new PravegaPollingOperationConnection(context, PravegaUtil.getKeycloakCredentialsString(context)));
     }
 
     @Override
     public PravegaPollingManager createListenManager(ConnectorContext context) {
-        return new PravegaPollingManager(new PravegaPollingManagerConnection(context, PravegaUtil.checkAndSetCredentials(context, map)));
+        return new PravegaPollingManager(new PravegaPollingManagerConnection(context, PravegaUtil.getKeycloakCredentialsString(context)));
     }
 
     @Override