From b4361c5c3421a45633cda617962b1f1af7ad5157 Mon Sep 17 00:00:00 2001 From: Justin Baur <19896123+justindbaur@users.noreply.github.com> Date: Tue, 24 Sep 2024 16:39:59 -0400 Subject: [PATCH 1/6] Initial Database Testing Document --- docs/contributing/testing/database/index.md | 92 +++++++++++++++++++++ 1 file changed, 92 insertions(+) create mode 100644 docs/contributing/testing/database/index.md diff --git a/docs/contributing/testing/database/index.md b/docs/contributing/testing/database/index.md new file mode 100644 index 000000000..781648bd0 --- /dev/null +++ b/docs/contributing/testing/database/index.md @@ -0,0 +1,92 @@ +# Database Integration Testing + +Since Bitwarden has multiple database options (4), testing them all automatically is incredibly +important so that we don't have to have a full QA process for each database type. This is where the +`Infrastructure.IntegrationTest` project comes in, it allows setting up tests similarly to how the +databases are consumed in the application. Through their common interface, generally an +`I*Repository`. These tests are automatically ran on all supported databases in our testing +pipeline. + +## Creating a new test + +To create a new database test, just add the `[DatabaseTheory]` and `[DatabaseData]` attributes to +test. Now, in the parameters of the test you are able to "inject" any repository layer services +directly into the test. The test will run for every database that is +[configured in the current environment](#configure-the-tests). Since you inject the interface of the +service, some runs of the test will use the Dapper based repository implementation targeting +Microsoft SQL Server and others will use the Entity Framework Core based implementations, which we +use for MySql, Postgres, and SQLite. + +The goal of database tests is to test the business logic that is encapsulated in a given method. If +the stored procedure in SQL Server then calls another procedure to update the +`User.AccountRevisionDate` then the same EF implementation should do that as well. Then by running +the test against all variants, we are ensuring all the variants are feature equal. Locally, you may +want to only the SQL Server tests along with one EF implementation, SQLite is often the easiest. +This may work well for a very long time and save you some time overall but there are differences +between the EF database providers such that you will one day get errors in the CI pipeline. + +## Configure the databases + +The databases are expected to have the latest migrations applied to them. + +## Configure the tests + +The tests are configured through +[.NET Configuration](https://learn.microsoft.com/en-us/dotnet/core/extensions/configuration). In the +order they are applied, user secrets, environment variables prefixed with `BW_TEST_`, and command +line args. + +```C# +public class Root +{ + public Database[] Databases { get; set; } +} + +public class Database +{ + public SupportedDatabaseProviders Type { get; set; } + public string ConnectionString { get; set; } = default!; + public bool UseEf { get; set; } + public bool Enabled { get; set; } = true; +} +``` + +The `Type` property is an enum with the supported values being `SqlServer`, `MySql`, `Postgres`, or +`Sqlite`. The `UseEf` property is only utilized it the `Type` is set to `SqlServer`, by default +`SqlServer` will be configured with the Dapper repositories but by setting `UseEf` to `true`, it +will be configured with the Entity Framework Core repositories. `Enabled` allows you to easily +disable one database but not delete the entry, it can be helpful if you are encountering a problem +with just a single database type and want to run the tests just for it instead of for all of them. + +### Locally + +To set the tests up locally you may want to add the configuration to your `dev/secrets.json` file. +An example entry you might add is: + +```json +{ + ...other config... + "databases": [ + { + "type": "SqlServer", + "connectionString": "myConnectionString + } + ] +} +``` + +You can also configure the tests just like the pipeline, read more below. + +### Pipeline + +The database tests have been pre-configured to run on all supported databases in the +[`test-database.yml`](https://github.com/bitwarden/server/blob/main/.github/workflows/test-database.yml) +([permalink](https://github.com/bitwarden/server/blob/f7bc5dfb2ea31ca7b4c36238295cdcc4008ad958/.github/workflows/test-database.yml)) +file. + +The pipeline uses environment variables, an example entry you might add is: + +```bash +BW_TEST_DATABASES__0__TYPE: SqlServer +BW_TEST_DATABASES__0__CONNECTIONSTRING: myConnectionString +``` From 135886086f0ebbd77c6ed308ee97c6f767d8735d Mon Sep 17 00:00:00 2001 From: Justin Baur <19896123+justindbaur@users.noreply.github.com> Date: Wed, 16 Oct 2024 17:02:35 -0400 Subject: [PATCH 2/6] Fix JSON --- docs/contributing/testing/database/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/testing/database/index.md b/docs/contributing/testing/database/index.md index 781648bd0..fade4a03e 100644 --- a/docs/contributing/testing/database/index.md +++ b/docs/contributing/testing/database/index.md @@ -69,7 +69,7 @@ An example entry you might add is: "databases": [ { "type": "SqlServer", - "connectionString": "myConnectionString + "connectionString": "myConnectionString" } ] } From 1725368655a143e9a955e843a7ef2477ca71e5ab Mon Sep 17 00:00:00 2001 From: Justin Baur <19896123+justindbaur@users.noreply.github.com> Date: Thu, 21 Nov 2024 09:06:45 -0500 Subject: [PATCH 3/6] Apply suggestions from code review Co-authored-by: Matt Bishop --- docs/contributing/testing/database/index.md | 24 ++++++++++----------- 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/contributing/testing/database/index.md b/docs/contributing/testing/database/index.md index fade4a03e..a764458e0 100644 --- a/docs/contributing/testing/database/index.md +++ b/docs/contributing/testing/database/index.md @@ -2,14 +2,14 @@ Since Bitwarden has multiple database options (4), testing them all automatically is incredibly important so that we don't have to have a full QA process for each database type. This is where the -`Infrastructure.IntegrationTest` project comes in, it allows setting up tests similarly to how the -databases are consumed in the application. Through their common interface, generally an -`I*Repository`. These tests are automatically ran on all supported databases in our testing +`Infrastructure.IntegrationTest` project comes in, allowing the setup of tests similarly to how the +databases are consumed in the applications through their common interfaces, generally an +`I*Repository`. These tests are automatically executed on all supported databases in our testing pipeline. ## Creating a new test -To create a new database test, just add the `[DatabaseTheory]` and `[DatabaseData]` attributes to +To create a new database test just add the `[DatabaseTheory]` and `[DatabaseData]` attributes to test. Now, in the parameters of the test you are able to "inject" any repository layer services directly into the test. The test will run for every database that is [configured in the current environment](#configure-the-tests). Since you inject the interface of the @@ -32,11 +32,11 @@ The databases are expected to have the latest migrations applied to them. ## Configure the tests The tests are configured through -[.NET Configuration](https://learn.microsoft.com/en-us/dotnet/core/extensions/configuration). In the -order they are applied, user secrets, environment variables prefixed with `BW_TEST_`, and command +[.NET Configuration](https://learn.microsoft.com/en-us/dotnet/core/extensions/configuration) in the +order they are applied: user secrets, environment variables prefixed with `BW_TEST_`, and command line args. -```C# +```csharp public class Root { public Database[] Databases { get; set; } @@ -52,10 +52,10 @@ public class Database ``` The `Type` property is an enum with the supported values being `SqlServer`, `MySql`, `Postgres`, or -`Sqlite`. The `UseEf` property is only utilized it the `Type` is set to `SqlServer`, by default -`SqlServer` will be configured with the Dapper repositories but by setting `UseEf` to `true`, it +`Sqlite`. The `UseEf` property is only utilized if the `Type` is set to `SqlServer`; by default +`SqlServer` will be configured with the Dapper repositories, however by setting `UseEf` to `true` it will be configured with the Entity Framework Core repositories. `Enabled` allows you to easily -disable one database but not delete the entry, it can be helpful if you are encountering a problem +disable one database but not delete the entry; it can be helpful if you are encountering a problem with just a single database type and want to run the tests just for it instead of for all of them. ### Locally @@ -75,7 +75,7 @@ An example entry you might add is: } ``` -You can also configure the tests just like the pipeline, read more below. +You can also configure the tests just like the pipeline. ### Pipeline @@ -84,7 +84,7 @@ The database tests have been pre-configured to run on all supported databases in ([permalink](https://github.com/bitwarden/server/blob/f7bc5dfb2ea31ca7b4c36238295cdcc4008ad958/.github/workflows/test-database.yml)) file. -The pipeline uses environment variables, an example entry you might add is: +The pipeline uses environment variables. An example entry you might add is: ```bash BW_TEST_DATABASES__0__TYPE: SqlServer From 6c4c99ebd2ca6a698c18825a818e836942d0dc22 Mon Sep 17 00:00:00 2001 From: Justin Baur <19896123+justindbaur@users.noreply.github.com> Date: Thu, 21 Nov 2024 09:12:33 -0500 Subject: [PATCH 4/6] Apply suggestions from code review Co-authored-by: Matt Bishop --- docs/contributing/testing/database/index.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/contributing/testing/database/index.md b/docs/contributing/testing/database/index.md index a764458e0..c82a06ae6 100644 --- a/docs/contributing/testing/database/index.md +++ b/docs/contributing/testing/database/index.md @@ -19,15 +19,15 @@ use for MySql, Postgres, and SQLite. The goal of database tests is to test the business logic that is encapsulated in a given method. If the stored procedure in SQL Server then calls another procedure to update the -`User.AccountRevisionDate` then the same EF implementation should do that as well. Then by running -the test against all variants, we are ensuring all the variants are feature equal. Locally, you may -want to only the SQL Server tests along with one EF implementation, SQLite is often the easiest. +`User.AccountRevisionDate` then the same EF implementation should do that as well. By running +the test against all variants we are ensuring all the variants are feature-equal. Locally, you may +want to only run the SQL Server tests along with one EF implementation; SQLite is often the easiest in that situation. This may work well for a very long time and save you some time overall but there are differences between the EF database providers such that you will one day get errors in the CI pipeline. ## Configure the databases -The databases are expected to have the latest migrations applied to them. +The databases are expected to have the latest migrations applied. ## Configure the tests From 7cd0f8a0d4d04614b3dfb222095ae70b11c39760 Mon Sep 17 00:00:00 2001 From: Justin Baur <19896123+justindbaur@users.noreply.github.com> Date: Thu, 21 Nov 2024 09:13:35 -0500 Subject: [PATCH 5/6] Prettier --- docs/contributing/testing/database/index.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/contributing/testing/database/index.md b/docs/contributing/testing/database/index.md index c82a06ae6..38b86989b 100644 --- a/docs/contributing/testing/database/index.md +++ b/docs/contributing/testing/database/index.md @@ -19,11 +19,12 @@ use for MySql, Postgres, and SQLite. The goal of database tests is to test the business logic that is encapsulated in a given method. If the stored procedure in SQL Server then calls another procedure to update the -`User.AccountRevisionDate` then the same EF implementation should do that as well. By running -the test against all variants we are ensuring all the variants are feature-equal. Locally, you may -want to only run the SQL Server tests along with one EF implementation; SQLite is often the easiest in that situation. -This may work well for a very long time and save you some time overall but there are differences -between the EF database providers such that you will one day get errors in the CI pipeline. +`User.AccountRevisionDate` then the same EF implementation should do that as well. By running the +test against all variants we are ensuring all the variants are feature-equal. Locally, you may want +to only run the SQL Server tests along with one EF implementation; SQLite is often the easiest in +that situation. This may work well for a very long time and save you some time overall but there are +differences between the EF database providers such that you will one day get errors in the CI +pipeline. ## Configure the databases From 5b13c27ec9ce0be1f8906e1596325a04e44f8de6 Mon Sep 17 00:00:00 2001 From: Justin Baur <19896123+justindbaur@users.noreply.github.com> Date: Thu, 21 Nov 2024 09:16:26 -0500 Subject: [PATCH 6/6] Remove `4` --- docs/contributing/testing/database/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/testing/database/index.md b/docs/contributing/testing/database/index.md index 38b86989b..0f78cf802 100644 --- a/docs/contributing/testing/database/index.md +++ b/docs/contributing/testing/database/index.md @@ -1,6 +1,6 @@ # Database Integration Testing -Since Bitwarden has multiple database options (4), testing them all automatically is incredibly +Since Bitwarden has multiple database options, testing them all automatically is incredibly important so that we don't have to have a full QA process for each database type. This is where the `Infrastructure.IntegrationTest` project comes in, allowing the setup of tests similarly to how the databases are consumed in the applications through their common interfaces, generally an