Skip to content

Commit

Permalink
SOLR-14115: Update the docs to reference to the bin/solr equivalent o…
Browse files Browse the repository at this point in the history 
…f the zkcli.sh (#2428)

Didn't eliminate zookeeper-utilities.adoc as there may be an option to break up the solr-control-script-reference.adoc that became apparent in this commit, however saving for another day.
  • Loading branch information
@epugh
epugh authored Apr 30, 2024
1 parent 963b629 commit b530aa5
Show file tree
Hide file tree
Showing 4 changed files with 21 additions and 203 deletions.
2 changes: 1 addition & 1 deletion ...uide/modules/deployment-guide/pages/monitoring-with-prometheus-and-grafana.adoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -232,7 +232,7 @@ Extra JVM options.
|===
+
Credentials for connecting to a ZooKeeper host that is protected with ACLs.
For more information on what to include in this variable, refer to the section xref:zookeeper-access-control.adoc#zookeeper-acls-in-solr-scripts[ZooKeeper ACLs in Solr Scripts] or the example <<getting-metrics-from-a-secured-solrcloud>> below.
For more information on what to include in this variable, refer to the section xref:zookeeper-access-control.adoc#zookeeper-acls-in-solr-cli[ZooKeeper ACLs in Solr CLI] or the example <<getting-metrics-from-a-secured-solrcloud>> below.

`CLASSPATH_PREFIX`::
+
Expand Down
6 changes: 0 additions & 6 deletions solr/solr-ref-guide/modules/deployment-guide/pages/taking-solr-to-production.adoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -278,12 +278,6 @@ We can use the mkroot command for that:
bin/solr zk mkroot /solr -z <ZK_node>:<ZK_PORT>
----

[NOTE]
====
If you also want to bootstrap ZooKeeper with existing `solr_home`, you can instead use the `zkcli.sh` / `zkcli.bat` `bootstrap` command, which will also create the chroot path if it does not exist.
See xref:zookeeper-utilities.adoc[] for more info.
====

==== Unknown core deletion

When Solr loads a core from a filesystem it will check for corresponding cluster state in ZooKeeper. If no corresponding entry exists the core will be skipped and a warning will be logged.
Expand Down
108 changes: 6 additions & 102 deletions solr/solr-ref-guide/modules/deployment-guide/pages/zookeeper-access-control.adoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ The credentials are used to get permission to perform operations in ZooKeeper.
. Control which ACLs Solr will add to znodes (ZooKeeper files/folders) it creates in ZooKeeper.
. Control it "from the outside", so that you do not have to modify and/or recompile Solr code to turn this on.

Solr nodes, clients, and tools (e.g., ZkCLI) always use a java class called {solr-javadocs}/solrj-zookeeper/org/apache/solr/common/cloud/SolrZkClient.html[`SolrZkClient`] to deal with their ZooKeeper stuff.
Solr nodes, clients, and tools (e.g., SolrCLI) always use a java class called {solr-javadocs}/solrj-zookeeper/org/apache/solr/common/cloud/SolrZkClient.html[`SolrZkClient`] to deal with their ZooKeeper stuff.
The implementation of the solution described here is all about changing `SolrZkClient`.
If you use `SolrZkClient` in your application, the descriptions below will be true for your application too.

Expand Down Expand Up @@ -269,19 +269,6 @@ SOLR_ZK_CREDS_AND_ACLS="-DzkACLProvider=org.apache.solr.common.cloud.DigestZkACL
-DzkDigestUsername=admin-user -DzkDigestPassword=CHANGEME-ADMIN-PASSWORD \
-DzkDigestReadonlyUsername=readonly-user -DzkDigestReadonlyPassword=CHANGEME-READONLY-PASSWORD"
SOLR_OPTS="$SOLR_OPTS $SOLR_ZK_CREDS_AND_ACLS"
----
.zkcli.sh
[source,bash]
----
# Settings for ZK ACL
SOLR_ZK_CREDS_AND_ACLS="-DzkACLProvider=org.apache.solr.common.cloud.DigestZkACLProvider \
-DzkCredentialsProvider=org.apache.solr.common.cloud.DigestZkCredentialsProvider \
-DzkCredentialsInjector=org.apache.solr.common.cloud.VMParamsZkCredentialsInjector \
-DzkDigestUsername=admin-user -DzkDigestPassword=CHANGEME-ADMIN-PASSWORD \
-DzkDigestReadonlyUsername=readonly-user -DzkDigestReadonlyPassword=CHANGEME-READONLY-PASSWORD"
----
====

Expand All @@ -300,17 +287,6 @@ set SOLR_ZK_CREDS_AND_ACLS=-DzkACLProvider=org.apache.solr.common.cloud.DigestZk
-DzkDigestReadonlyUsername=readonly-user -DzkDigestReadonlyPassword=CHANGEME-READONLY-PASSWORD
set SOLR_OPTS=%SOLR_OPTS% %SOLR_ZK_CREDS_AND_ACLS%
----
.zkcli.bat
[source,powershell]
----
REM Settings for ZK ACL
set SOLR_ZK_CREDS_AND_ACLS=-DzkACLProvider=org.apache.solr.common.cloud.DigestZkACLProvider ^
-DzkCredentialsProvider=org.apache.solr.common.cloud.DigestZkCredentialsProvider ^
-DzkCredentialsInjector=org.apache.solr.common.cloud.VMParamsZkCredentialsInjector ^
-DzkDigestUsername=admin-user -DzkDigestPassword=CHANGEME-ADMIN-PASSWORD ^
-DzkDigestReadonlyUsername=readonly-user -DzkDigestReadonlyPassword=CHANGEME-READONLY-PASSWORD
----
====
--

Expand Down Expand Up @@ -347,17 +323,6 @@ SOLR_ZK_CREDS_AND_ACLS="-DzkACLProvider=org.apache.solr.common.cloud.DigestZkACL
SOLR_OPTS="$SOLR_OPTS $SOLR_ZK_CREDS_AND_ACLS"
----
.zkcli.sh
[source,bash]
----
# Settings for ZK ACL
SOLR_ZK_CREDS_AND_ACLS="-DzkACLProvider=org.apache.solr.common.cloud.DigestZkACLProvider \
-DzkCredentialsProvider=org.apache.solr.common.cloud.DigestZkCredentialsProvider \
-DzkCredentialsInjector=org.apache.solr.common.cloud.VMParamsZkCredentialsInjector \
-DzkDigestCredentialsFile=SOLR_HOME_DIR/server/etc/zookeepercredentials.properties"
----
====

[example.tab-pane#file-system-props-windows]
Expand All @@ -374,16 +339,6 @@ set SOLR_ZK_CREDS_AND_ACLS=-DzkACLProvider=org.apache.solr.common.cloud.DigestZk
-DzkDigestCredentialsFile=SOLR_HOME_DIR/server/etc/zookeepercredentials.properties
set SOLR_OPTS=%SOLR_OPTS% %SOLR_ZK_CREDS_AND_ACLS%
----
.zkcli.bat
[source,powershell]
----
REM Settings for ZK ACL
set SOLR_ZK_CREDS_AND_ACLS=-DzkACLProvider=org.apache.solr.common.cloud.DigestZkACLProvider ^
-DzkCredentialsProvider=org.apache.solr.common.cloud.DigestZkCredentialsProvider ^
-DzkCredentialsInjector=org.apache.solr.common.cloud.VMParamsZkCredentialsInjector ^
-DzkDigestCredentialsFile=SOLR_HOME_DIR/server/etc/zookeepercredentials.properties
----
====
--

Expand Down Expand Up @@ -413,17 +368,6 @@ SOLR_ZK_CREDS_AND_ACLS="-DzkACLProvider=org.apache.solr.common.cloud.DigestZkACL
-DzkCredentialsProvider=org.apache.solr.common.cloud.DigestZkCredentialsProvider \
-DzkCredentialsInjector=fully.qualified.class.CustomInjectorClassName"
SOLR_OPTS="$SOLR_OPTS $SOLR_ZK_CREDS_AND_ACLS"
----
.zkcli.sh
[source,bash]
----
# Settings for ZK ACL
SOLR_ZK_CREDS_AND_ACLS="-DzkACLProvider=org.apache.solr.common.cloud.DigestZkACLProvider \
-DzkCredentialsProvider=org.apache.solr.common.cloud.DigestZkCredentialsProvider \
-DzkCredentialsInjector=fully.qualified.class.CustomInjectorClassName
----
====

Expand All @@ -440,30 +384,12 @@ set SOLR_ZK_CREDS_AND_ACLS=-DzkACLProvider=org.apache.solr.common.cloud.DigestZk
-DzkCredentialsInjector=fully.qualified.class.CustomInjectorClassName
set SOLR_OPTS=%SOLR_OPTS% %SOLR_ZK_CREDS_AND_ACLS%
----
.zkcli.bat
[source,powershell]
----
REM Settings for ZK ACL
set SOLR_ZK_CREDS_AND_ACLS=-DzkACLProvider=org.apache.solr.common.cloud.DigestZkACLProvider ^
-DzkCredentialsProvider=org.apache.solr.common.cloud.DigestZkCredentialsProvider ^
-DzkCredentialsInjector=fully.qualified.class.CustomInjectorClassName
----
====
--



=== ZooKeeper ACLs in Solr Scripts

There are two scripts that impact ZooKeeper ACLs:

* For *nix systems: `bin/solr` & `server/scripts/cloud-scripts/zkcli.sh`
* For Windows systems: `bin/solr.cmd` & `server/scripts/cloud-scripts/zkcli.bat`

[IMPORTANT]
Both the solr.in.* and the zkcli.* files will need to be updated with the same password for everything to work.
The contents may appear redundant, but the scripts will not consult each other during operations.
=== ZooKeeper ACLs in Solr CLI

These Solr scripts can enable use of ZooKeeper ACLs by setting the appropriate system properties.

Expand All @@ -489,17 +415,6 @@ in the following files:
# -DzkDigestReadonlyUsername=readonly-user -DzkDigestReadonlyPassword=CHANGEME-READONLY-PASSWORD"
#SOLR_OPTS="$SOLR_OPTS $SOLR_ZK_CREDS_AND_ACLS"
----
.zkcli.sh
[source,bash]
----
# Settings for ZK ACL
#SOLR_ZK_CREDS_AND_ACLS="-DzkACLProvider=org.apache.solr.common.cloud.DigestZkACLProvider \
# -DzkCredentialsProvider=org.apache.solr.common.cloud.DigestZkCredentialsProvider \
# -DzkCredentialsInjector=org.apache.solr.common.cloud.VMParamsZkCredentialsInjector \
# -DzkDigestUsername=admin-user -DzkDigestPassword=CHANGEME-ADMIN-PASSWORD \
# -DzkDigestReadonlyUsername=readonly-user -DzkDigestReadonlyPassword=CHANGEME-READONLY-PASSWORD"
----
====

[example.tab-pane#windows]
Expand All @@ -517,17 +432,6 @@ REM -DzkDigestUsername=admin-user -DzkDigestPassword=CHANGEME-ADMIN-PASSWORD ^
REM -DzkDigestReadonlyUsername=readonly-user -DzkDigestReadonlyPassword=CHANGEME-READONLY-PASSWORD
REM set SOLR_OPTS=%SOLR_OPTS% %SOLR_ZK_CREDS_AND_ACLS%
----
.zkcli.bat
[source,powershell]
----
REM Settings for ZK ACL
REM set SOLR_ZK_CREDS_AND_ACLS=-DzkACLProvider=org.apache.solr.common.cloud.DigestZkACLProvider ^
REM -DzkCredentialsProvider=org.apache.solr.common.cloud.DigestZkCredentialsProvider ^
REM -DzkCredentialsInjector=org.apache.solr.common.cloud.VMParamsZkCredentialsInjector ^
REM -DzkDigestUsername=admin-user -DzkDigestPassword=CHANGEME-ADMIN-PASSWORD ^
REM -DzkDigestReadonlyUsername=readonly-user -DzkDigestReadonlyPassword=CHANGEME-READONLY-PASSWORD
----
====
--

Expand All @@ -536,8 +440,8 @@ REM -DzkDigestReadonlyUsername=readonly-user -DzkDigestReadonlyPassword=CHANGEM
Over the lifetime of operating your Solr cluster, you may decide to move from an unsecured ZooKeeper to a secured instance.
Changing the configured `zkACLProvider` in `solr.xml` will ensure that newly created nodes are secure, but will not protect the already existing data.

To modify all existing ACLs, you can use the `updateacls` command with Solr's ZkCLI.
First uncomment the `SOLR_ZK_CREDS_AND_ACLS` environment variable definition in `server/scripts/cloud-scripts/zkcli.sh` (or `zkcli.bat` on Windows) and fill in the passwords for the admin-user and the readonly-user as described above in <<ZooKeeper ACLs in Solr Scripts>>.
To modify all existing ACLs, you can use the `bin/solr zk updateacls` command with Solr's CLI.
First uncomment the `SOLR_ZK_CREDS_AND_ACLS` environment variable definition in `solr.in.*` and fill in the passwords for the admin-user and the readonly-user as described above in <<ZooKeeper ACLs in Solr CLI>>.

Then run the command below appropriate for your operating system:

Expand Down Expand Up @@ -565,7 +469,7 @@ $ bin/solr.cmd zk updateacls /zk-path
Changing ACLs in ZooKeeper should only be done while your SolrCloud cluster is stopped.
Attempting to do so while Solr is running may result in inconsistent state and some nodes becoming inaccessible.

The VM properties `zkCredentialsInjector`, `zkACLProvider` and `zkCredentialsProvider`, included in the `SOLR_ZK_CREDS_AND_ACLS` environment variable in `zkcli.sh/.bat`, control the conversion:
The VM properties `zkCredentialsInjector`, `zkACLProvider` and `zkCredentialsProvider`, included in the `SOLR_ZK_CREDS_AND_ACLS` environment variable in `solr.in.*`, control the conversion:

* The Credentials Injector reads the credentials and pass them to the Credentials Provider.
When omitted, the process will use no credentials (suitable for an unsecure configuration).
Expand All @@ -574,4 +478,4 @@ When omitted, the process will use no credentials (suitable for an unsecure conf
* The ACL Provider will be used to compute the new ACLs.
When omitted, the process will set all permissions to all users, removing any security present.

The uncommented `SOLR_ZK_CREDS_AND_ACLS` environment variable in `zkcli.sh/.bat` sets the credentials and ACL providers to the `VMParamsZkCredentialsInjector`, `DigestZkCredentialsProvider` and `DigestZkACLProvider` implementations, described earlier in the page.
The uncommented `SOLR_ZK_CREDS_AND_ACLS` environment variable in `solr.in.*` sets the credentials and ACL providers to the `VMParamsZkCredentialsInjector`, `DigestZkCredentialsProvider` and `DigestZkACLProvider` implementations, described earlier in the page.
108 changes: 14 additions & 94 deletions solr/solr-ref-guide/modules/deployment-guide/pages/zookeeper-utilities.adoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,132 +16,51 @@
// specific language governing permissions and limitations
// under the License.

A ZooKeeper Command Line Interface (CLI) script is available to allow you to interact directly with Solr configuration files stored in ZooKeeper.
The Solr Command Line Interface (CLI) script is available to allow you to interact directly with Solr configuration files stored in ZooKeeper.

While Solr's Admin UI includes pages dedicated to the state of your SolrCloud cluster, it does not allow you to download or modify related configuration files.

TIP: See the section xref:cloud-screens.adoc[] for more information about using the Admin UI screens.

The ZooKeeper CLI script found in `server/scripts/cloud-scripts` let you upload configuration information to ZooKeeper.
It also provides a few other commands that let you link collection sets to collections, make ZooKeeper paths or clear them, and download configurations from ZooKeeper to the local filesystem.

Many of the functions provided by the `zkCli.sh` script are also provided by the xref:solr-control-script-reference.adoc[Solr Control Script], which may be more familiar as the start script ZooKeeper maintenance commands are very similar to Unix commands.
The ZooKeeper specific commands are provided by the xref:solr-control-script-reference.adoc[Solr Control Script].

.Solr's zkcli.sh vs ZooKeeper's zkCli.sh
[IMPORTANT]
====
The `zkcli.sh` provided by Solr is not the same as the https://zookeeper.apache.org/doc/current/zookeeperStarted.html#sc_ConnectingToZooKeeper[`zkCli.sh` included in ZooKeeper distributions].
The deprecated `zkcli.sh` provided by Solr is not the same as the https://zookeeper.apache.org/doc/current/zookeeperStarted.html#sc_ConnectingToZooKeeper[`zkCli.sh` included in ZooKeeper distributions].
ZooKeeper's `zkCli.sh` provides a completely general, application-agnostic shell for manipulating data in ZooKeeper.
Solr's `zkcli.sh` – discussed in this section – is specific to Solr, and has command line arguments specific to dealing with Solr data in ZooKeeper.
`zkcli.sh` has been deprecated in favour of the `bin/solr` commands which should be used instead.
`zkcli.sh` has been deprecated in favour of the `bin/solr` equivalent commands which should be used instead. It is no longer referenced in the Reference Guide.
====

== Using Solr's ZooKeeper CLI

Use the `help` option to get a list of available commands from the script itself, as in `./server/scripts/cloud-scripts/zkcli.sh help`.

Both `zkcli.sh` (for Unix environments) and `zkcli.bat` (for Windows environments) support the following command line options:

`-cmd <arg>`::
The CLI Command to be executed.
This parameter is *mandatory*.
The following commands are supported:

* `bootstrap`
* `upconfig`
* `downconfig`
* `linkconfig`
* `makepath`
* `get` and `getfile`
* `put` and `putfile`
* `clear`
* `list`
* `ls`
* `clusterprop`

`-z` or `-zkhost <locations>`::
ZooKeeper host address.
This parameter is *mandatory* for all CLI commands.

`-c` or `-collection <name>`::
For `linkconfig`: name of the collection.

`-d` or `-confdir <path>`::
For `upconfig`: a directory of configuration files.
For downconfig: the destination of files pulled from ZooKeeper

`-h` or `-help`::
Display help text.

`-n` or `-confname <arg>`::
For `upconfig`, `linkconfig`, `downconfig`: name of the configuration set.

`-r` or `-runzk <port>`::
Run ZooKeeper internally by passing the Solr run port; only for clusters on one machine.

`-s` or `-solrhome <path>`:: For `bootstrap` or when using `-runzk`: the *mandatory* solrhome location.

`-name <name>`::
For `clusterprop`: the *mandatory* cluster property name.

`-val <value>`::
For `clusterprop`: the cluster property value.
If not specified, *null* will be used as value.

[TIP]
====
The short form parameter options may be specified with a single dash (e.g., `-c mycollection`).
The long form parameter options may be specified using either a single dash (e.g., `-collection mycollection`) or a double dash (e.g., `--collection mycollection`)
====

== ZooKeeper CLI Examples
Use the `help` option to get a list of available ZooKeeper specifc commands from the script itself, as in `bin/solr zk -h`.

Below are some examples of using the `zkcli.sh` CLI which assume you have already started the SolrCloud example (`bin/solr start -e cloud -noprompt`)
== Solr CLI Examples

If you are on Windows machine, simply replace `zkcli.sh` with `zkcli.bat` in these examples.
Below are some examples of using the `bin/solr` CLI which assume you have already started the SolrCloud example (`bin/solr start -e cloud -noprompt`)

=== Upload a Configuration Directory

[source,bash]
----
./server/scripts/cloud-scripts/zkcli.sh -zkhost 127.0.0.1:9983 -cmd upconfig -confname my_new_config -confdir server/solr/configsets/_default/conf
----

=== Bootstrap ZooKeeper from an Existing solr.home

[source,bash]
----
./server/scripts/cloud-scripts/zkcli.sh -zkhost 127.0.0.1:2181 -cmd bootstrap -solrhome /var/solr/data
----

.Bootstrap with chroot
[NOTE]
====
Using the boostrap command with a ZooKeeper chroot in the `-zkhost` parameter, e.g., `-zkhost 127.0.0.1:2181/solr`, will automatically create the chroot path before uploading the configs.
====

=== Put Arbitrary Data into a New ZooKeeper file

[source,bash]
----
./server/scripts/cloud-scripts/zkcli.sh -zkhost 127.0.0.1:9983 -cmd put /my_zk_file.txt 'some data'
bin/solr zk upconfig -z 127.0.0.1:9983 -n my_new_config -d server/solr/configsets/_default/conf
----

=== Put a Local File into a New ZooKeeper File

[source,bash]
----
./server/scripts/cloud-scripts/zkcli.sh -zkhost 127.0.0.1:9983 -cmd putfile /my_zk_file.txt /tmp/my_local_file.txt
./bin/solr zk cp ./my_local_file.txt zk:/my_zk_file.txt -z 127.0.0.1:9983
----

=== Link a Collection to a Configset

[source,bash]
----
./server/scripts/cloud-scripts/zkcli.sh -zkhost 127.0.0.1:9983 -cmd linkconfig -collection gettingstarted -confname my_new_config
./bin/solr zk linkconfig --confname my_new_config -c gettingstarted -z 127.0.0.1:9983
----

=== Create a New ZooKeeper Path
Expand All @@ -150,17 +69,18 @@ This can be useful to create a chroot path in ZooKeeper before first cluster sta

[source,bash]
----
./server/scripts/cloud-scripts/zkcli.sh -zkhost 127.0.0.1:2181 -cmd makepath /solr
bin/solr zk mkroot /solr -z 127.0.0.1:9983
----

=== Set a Cluster Property

This command will add or modify a single cluster property in `clusterprops.json`.
Use this command instead of the usual getfile -> edit -> putfile cycle.

Unlike the xref:cluster-node-management.adoc#clusterprop[CLUSTERPROP] command on the Collections API, this command does *not* require a running Solr cluster.

[source,bash]
----
./server/scripts/cloud-scripts/zkcli.sh -zkhost 127.0.0.1:2181 -cmd clusterprop -name urlScheme -val https
bin/solr cluster --property urlScheme --value https -z 127.0.0.1:9983
----

NOTE: This command is not nested under `bin/solr zk` like the others.

0 comments on commit b530aa5

Please sign in to comment.