Solr 12276 angularjs to angular migration

[NikolaiJDev]

https://issues.apache.org/jira/browse/SOLR-12276

Description

The fully simple migration to the newest version of Angular application which is Angular 18.x at this moment.

The basic idea of this migration it's not only legacy codebase migration and also support the modularity of the application for some third-party extensions. Migration from JavaScript to TypeScript opens and opportunity for in some sort of the Strong typed development which gives more opportunity to avoid some shadowed changes or assignment. As well as implement Interface - Adapter which allow to implement standalone components and combined with main application.

Solution

This is a draft version for the New codebase of Solr Ui application. In this PR there are only Fully functioned a Dashboard

Tests

Please describe the tests you've developed or run to confirm this patch implements the feature or solves the problem.

Checklist

Please review the following and check all that apply:

• I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
• I have created a Jira issue and added the issue ID to my pull request title.
• I have given Solr maintainers access to contribute to my PR branch. (optional but recommended, not available for branches on forks living under an organisation)
• I have developed this patch against the main branch.
• I have run ./gradlew check.
• I have added tests for my changes.
• I have added documentation for the Reference Guide

[NikolaiJDev]

JFYI: About 200 or more changes in files are moving to separate directory old kind of application and migration of images to new application. So actual changed files are less the 100 or even fewer. Please take into account, for other components the PRs won't be huge like this one. This happened just because setting up completely new framework from scratches.

[malliaridis]

First of all I want to thank you for the time and effort you have put together into this PR and migration. It is nice to have options to choose from and to compare with one another for decisions of this scale.

I have a few questions in regards to the changes included that would be great if you can answer them:

• What was the reason again you had to rename / move the initial webapp module?
• Have you looked into the file size of a deployment? This would be an interesting value to compare with other framework if you have it available.
• The current state of the PR seems to not integrate the Angular project into the current "deployment", resulting in CORS issues when running with npm start. Do you have any instructions on how to run it so that we can see the UI in action?

A short feedback about improving the quality of your PR: If it would be possible to keep the current webapp untouched, the PR would be much smaller and easier to review. The lines of code would also better represent the actual work and part that should be reviewed. Seeing 55k lines of code "scares" many reviewers, even if it is just generated or moved stuff.

Right now the files relevant for review are probably less than 60 files, all located in currentUi/solrUi.

I would also like to have a documentation that provide resources for further reading or basic ideas behind things like project structure (of the frontend of course). I think this would give our current community an easier start for both reviewing and working with the code.

Starting and testing the current UI seems also not to be working in this PR, which would be necessary for other reviewers to form an opinion.

Note: From here on, many of the things I mention are based on my own opinions and experiences.

From what I remember, in your migration you try to migrate the entire UI as is to Angular and then introduce changes in design and features. From a general perspective, I would follow this approach 90% of the times, as it quicker, less error-prone and often comes with test migrations, guaranteeing the functionality of the new code base. For our case however I believe that it is not worth the effort to study the existing code and try mitigating a behavior that is probably strongly outdated. At the time of writing of the UI I believe many concepts were not established and therefore many parts of the code may be implemented differently nowadays. I would therefore consider reevaluating the use cases / features to only "migrate" those useful to the users and directly implement them as it is expected by the framework.

About the actual code

What I saw and liked is that you included pipes, which is one of the ways Angular is processing and transforming data. This allows other reviewers unfamiliar with Angular to see some of its features.

What I also like and is included is the use of environments that allow the project to be built for development or production. We are already considering introducing such concepts in places like the CLI.

What I would probably do different is using Typescript optionals (like in value?: type) wherever possible and necessary, rather than relying on undefined (like in value: type | undefined`).

I'd also like to follow naming conventions and avoid mixing things. I saw that some files and folders are following the kebab-case with dot-notation, other use the PascalCase and some even use camelCase.

I am not sure if the styles you used are based on a UI kit, but I recommend using one as a library. This would require less customization with CSS, reducing the amount of code that needs maintenance even further.

When it comes to style values, I strongly encourage you to make use of the 8-point grid system.

I've also seen that each component's SCSS defines its own colors at the top. I would avoid that because it makes it harder to later update the look and feel. You would have to change multiple places for adjusting these values. Since you are using SCSS it shouldn't be a problem importing variables from another file.

What would also simplify and reduce the stylesheets, is to make heavier use of inline styling for styles that are applied to single elements and not groups / classes. For example, attributes like background-image could inline-styles of the element or be added as separate DOM elements rather than styles.

To reduce the influence of changing APIs, I would avoid using the same data entities from the API in the view. Introducing an abstraction layer and mapping the values would reduce the impact of API changes to only the services.

To avoid "repetitive" code in components, I would prefer using data binding. For simple getters like in main.component.ts one-way data binding, from source to view, would be more than sufficient and reduce a lot the sizes of components. And in combination with the previous recommendation of abstracting the API data classes from the rest of the app, it would also be safe to directly use the attributes from the component (as these values would be read-only in the UI).

About Angular

What I personally like about Angular is that it is based on Typescript and well defined. It provides everything you need in a web frontend and has great documentation. What I do not like is the use of annotations and generated code. This often results in unknown behavior and outcomes, sometimes even limitations.

Personally I am not a big fan of HTML and CSS. I have spent way too many hours with trying to position and size things correctly. The same styles applied to an element strongly depends on its parents' style, and therefore may look completely different if reused.

Now about the main topic here,

Using Angular for the Admin UI migration

Angular is a great framework that provides a great documentation. Like any other framework, Angular requires time to learn, and because it makes heavy use of annotations and uses a lot of design patterns, it will take more time to understand what is happening in the background. It is not difficult for our use case to learn, it only needs a more time compared to other frameworks.

Because Angular is a framework, the project structure is given and well defined with various patterns. Therefore, we do not need to be creative in that part and can simply follow the instructions / documentation of Angular.

When it comes to compatibility with previous source code, Angular does not provide any benefit over any other framework. AngularJS is a thing on its own and no matter what framework we would use for the migration, we still have to migrate the entire UI into a new codebase.

Reusing any of the existing code will be difficult and migrating it may be more error-prone than simply writing new fresh code based on the framework in use. I prefer bugs in code I have newly written / reviewed, rather than bugs in code I have altered and tried to match legacy behavior. In my opinion, we should use the existing code only for understanding what the UI was doing before.

Looking into our current challenges, one of the main problems is the maintenance of the current webapp. The current community is less experienced in frontend development and web technologies. This has proven to be a problem when it comes to maintain the old and new codebase. Therefore, taking the community's preferences is crucial for the long-term livability of the frontend.

Another point that should not be taken lightly here, is that an Angular frontend (and any other web-based framework) comes with its own package manager (NPM), languages (Javascript, Typescript, HTML) and stylesheets (CSS). Developers would need to familiarize themselves with new technologies besides the ones already in use. Integrating a web-based frontend into our current project is possible (already proven), but multiple workarounds and customizations are necessary. Choosing a technology that is based on the current stack (Gradle, JVM) would be beneficial for the community, simplify processes and reduce the amount of customizations / workarounds.

I believe that if we would want a desktop client in the future, we would need to make use of third-party solutions like Electron.

A few less important notes about the content of the PR

The below notes can be ignored and will become relevant only in case this PR will be chosen:

Multiple license headers have been altered

This should not happen for files that are not our own or generated. I saw that multiple CSS files and js files from the libraries have modified license headers. This probably happened while moving things around.

License headers added to or missing from some files

In HTML if I remember correct we avoided adding license headers due to the file sizes and additional traffic needed. If we plan to add them, we should make consider stripping them off from individual files and provide a single license in the packaged webapp (if that is an acceptable approach).

References to assets seems problematic

I believe that references of the type ../../../solrUi/src/assets/* are likely to break during deployment, as the project is being built and moved to different directories. If I am not mistaken, the src folder does also not exist as such once the project is built. I believe the correct reference would be just /assets/*. This may happend unintentionally while moving things around. Pointing that out in case we decide to follow this approach.

I have left some inline notes as well, sometimes repeating what I said above just for reference.

Offtopic

There has been a discussion in the Compose PR about assets and their representation. Luckily, the new UI designs focus on vector icons and do not use PNGs for assets, like it was done before. This is in my opinion very beneficial from the theming and maintenance perspectives. We should therefore avoid reusing the existing assets in a new UI.

I'd also like to see types and interfaces from our OpenAPI spec being generated and used in web projects, once available.

A web-based solution would probably require us to find a solution for integrating license / notice file checks related to the dependencies. I am not sure how this is handled right now, but we will probably require some checking to be done for all the packages loaded by web-projects. And we may end up with a custom solution if we can't find something that integrates well with our current stack. This topic is worth its own thread though.

[malliaridis]

I believe this doesn't have to be an angular module / component?

[NikolaiJDev]

This is part of modularity that Eric has talked. That other extensions can be applied on the fly, this is just one of the first step to provide common logic across multiple components and modules within Angular, as well in the future there will Interface/Adapter implemented to support external modules integration to the application.

[malliaridis]

I see. I saw only the static functions in the module and thought "if all its content is static, what is the role of this class to be a module?". I didn't have a context where this module could be used.

It is nice that you thought about extendability of the frontend, I was asked about this too. There are definitely multiple ways to achieve this, but I believe integrating it too early may overcomplicate things.

[NikolaiJDev]

@malliaridis, well If we begin our implementation and only at the later stage add the structure to integrate third party extension, it might require significant amount of work at the latest stage of the development.
It's simple for example, if we want to integrate multi-laguage support into our UI, it would be easier to add it at this stage of the application development rather at the latest stage.
Regarding this Shared module, is more about code grouping within the application structure.
At this point of the implementation, we have only one component is provided for review.

[NikolaiJDev]

First of all I want to thank you for the time and effort you have put together into this PR and migration. It is nice to have options to choose from and to compare with one another for decisions of this scale.

No problem, this is just an experience for me and my friends.

I have a few questions in regards to the changes included that would be great if you can answer them:

• What was the reason again you had to rename / move the initial webapp module?
• Have you looked into the file size of a deployment? This would be an interesting value to compare with other framework if you have it available.
• The current state of the PR seems to not integrate the Angular project into the current "deployment", resulting in CORS issues when running with npm start. Do you have any instructions on how to run it so that we can see the UI in action?

Let's start from the first question:

1. I've planned to integrate two completely different solution that is going to work in parallel and place switch to new implementation that person can simply change it to old one.
2. At this moment including all of the assets the total solution is cost 267 kb, in the later versions it might grow unto 2 - 3 mb may be a little bit more, because we are producing a bundle that holds everything.
3. That's correct, I didn't invest enough time to merge to two different approaches to work under same server, usually for that is responsible app server in our case, it must be done via web.xml + might be require additional filter or something changes in the java app itself to consume and work with two approaches.

A short feedback about improving the quality of your PR: If it would be possible to keep the current webapp untouched, the PR would be much smaller and easier to review. The lines of code would also better represent the actual work and part that should be reviewed. Seeing 55k lines of code "scares" many reviewers, even if it is just generated or moved stuff.

Okay, I'll try to revert most of the changes related to the current solution back that it won't affect PR.

I would also like to have a documentation that provide resources for further reading or basic ideas behind things like project structure (of the frontend of course). I think this would give our current community an easier start for both reviewing and working with the code.

Well, I'm not sure what do you mean under the documentation with this. If you're speaking about the documentation such as README file then I will add one how to start local environment, until we'll have complete integration with backend solution, it won't help. I hope I correctly understood you.

Starting and testing the current UI seems also not to be working in this PR, which would be necessary for other reviewers to form an opinion.

If you're speaking about unit/integration test, first of all I'm planning to move all the current functionality that exists at this moment toward to the new implementation that it went smoothly, and we can simply continue add features and refactor the current solution.

If you're speaking about the way how to start and poke review it. Then I do everything from package.json until we integrated it with current backend solution.

[NikolaiJDev]

References to assets seems problematic

I believe that references of the type ../../../solrUi/src/assets/* are likely to break during deployment, as the project is being built and moved to different directories. If I am not mistaken, the src folder does also not exist as such once the project is built. I believe the correct reference would be just /assets/*. This may happend unintentionally while moving things around. Pointing that out in case we decide to follow this approach.

I won't be so worry about that, this is just for source while webpack that runs under the hood of the ng build, packs all that requires to assets of the apllication here is an example:

#environment {
  background-image: url("./media/box-DCQ6CIJF.png");

I took this line from the compiled and minimized css file that appeared after compilation.
We only need to worry where and how to place those images at this moment that simplify our development process.

[malliaridis]

The current state of the PR seems to not integrate the Angular project into the current "deployment", resulting in CORS issues when running with npm start. Do you have any instructions on how to run it so that we can see the UI in action?

3. That's correct, I didn't invest enough time to merge to two different approaches to work under same server, usually for that is responsible app server in our case, it must be done via web.xml + might be require additional filter or something changes in the java app itself to consume and work with two approaches.

Thanks. So that would still be an open TODO in case we decide to proceed with this one.

A short feedback about improving the quality of your PR: [...]

Okay, I'll try to revert most of the changes related to the current solution back that it won't affect PR.

This now way easier to review, thanks a lot.

I would also like to have a documentation that provide resources for further reading or basic ideas behind things like project structure (of the frontend of course). I think this would give our current community an easier start for both reviewing and working with the code.

Well, I'm not sure what do you mean under the documentation with this. If you're speaking about the documentation such as README file then I will add one how to start local environment, until we'll have complete integration with backend solution, it won't help. I hope I correctly understood you.

Yes, what I (and probably others as well) would like to see is how it would integrate into the current project (which is as described above an open TODO), how someone would work and test the code base (like running the UI with npm ng serve), and where the webapp can be accessed / seen (like at localhost:3000). I believe the module's / angular project's README file is the best place to tell other developers how to get started with this sub-project. Angular already provides some basic information, this could be extended.

Starting and testing the current UI seems also not to be working in this PR, which would be necessary for other reviewers to form an opinion.

[...]
If you're speaking about the way how to start and poke review it. Then I do everything from package.json until we integrated it with current backend solution.

I meant starting and poking around, yes. So there is no state yet that makes request to a Solr backend, is that correct?

I think you removed a few too manyfiles in your latest updates. The environment files under /src/environments are missing and causing an error when launching the app with ng serve.

And one last tip, it's fine and also recommended to include package-lock.json in the VCS, as it ensures reproducable states of the project. :)

[NikolaiJDev]

Yes, what I (and probably others as well) would like to see is how it would integrate into the current project (which is as described above an open TODO), how someone would work and test the code base (like running the UI with npm ng serve), and where the webapp can be accessed / seen (like at localhost:3000). I believe the module's / angular project's README file is the best place to tell other developers how to get started with this sub-project. Angular already provides some basic information, this could be extended.

It will be integrated same way as it was done before because it has to be shipped together with main application then using Gradle service is a great practice, you asked me to be waved out current application from new one, so I removed the concept of integration for time being. However, the refactoring of the Gradle job and few changes to the web.xml might be required to provide to fully support the current approach. As I've already mentioned to you previously - It might be also required to make some tweaks and changes in the Backend filter, what we can only understand on the stage of integration, I hope it won't be required, but It may.

I meant starting and poking around, yes. So there is no state yet that makes request to a Solr backend, is that correct?

I think you removed a few too manyfiles in your latest updates. The environment files under /src/environments are missing and causing an error when launching the app with ng serve.

And one last tip, it's fine and also recommended to include package-lock.json in the VCS, as it ensures reproducable states of the project. :)

Well, git shows me that envs are in place, I trigger the update as well for that.
As well, I want to inform to run correctly app, you need to have either docker container with working solr application or it must run from the same hostname, just for clarification localhost that you are running from isn't same as for backend, CORS will appear to avoid it you need to make a proxy configuration such as Nginx server which will locate UI and middle tier application under same umbrella scope. It will help browser to avoid this issue. Because we will emulate same patter as if it will be run from same space or machine as backend application.

[NikolaiJDev]

Regarding updating of the Gradle script, I'm looking for the solution that will easily integrate Angular build process into it, I found some of them, but they are outdated.