Giter VIP home page Giter VIP logo

shogun's Introduction

SHOGun Logo SHOGun

publish test

SHOGun is an application framework written in Java for building scaleable web application backends in the context of geospatial data infrastructures. It can be used directly without any specific customizations or highly customized to meet the demands of flexible project requirements.

In a nutshell

SHOGun…

  • …is written in Java and supports Java >= 21.
  • …is based on top of Spring / Spring Boot.
  • …provides a set of configuration entities to manage the contents of a geospatial data infrastructure (e.g. layers, applications, users or groups).
  • …provides (secured) web interfaces (REST & GraphQL) for accessing and manipulating these entities (e.g. for creating an application configuration).
  • …separates its functionalities into isolated microservices (e.g. for proxying OGC requests) and is highly scalable.

Development

Requirements

For the development of SHOGun the following tools are required locally:

Steps for development setup (for the first checkout)

To set up a local development setup, please proceed as follows:

  1. Checkout this repository:

    git clone [email protected]:terrestris/shogun.git
  2. Checkout the shogun-docker repository:

    git clone [email protected]:terrestris/shogun-docker.git
  3. Create a new project in IntelliJ by importing the first module:

    • File -> New -> Project from Existing Sources…
    • Navigate to the checkout of shogun
    • Select the Project Object Model file (pom.xml) of shogun
  4. Optional: You may also want to import the shogun-docker project. If so, import the folder as a module:

    • File -> New -> Module from Existing Sources…
    • Navigate to checkout of shogun-docker and choose the directory
  5. If not already done, the annotation processing of the Lombok plugin must be enabled. Check the settings for Lombok (Enable Lombok plugin for this project) and Annotation Processors (Enable annotation processing).

  6. Set up the shogun-docker requirements as described in here.

  7. Startup the containers (in the shogun-docker checkout directory):

    docker compose up
  8. The application is now available at https://localhost/.

Quick startup

If you already have a local development setup, just proceed as follows:

  1. Startup the containers (in the shogun-docker checkout directory):

    docker compose up
  2. The application is now available at https://localhost/.

Development hints

Local install

Install a new version of SHOGun to your local maven repository with

mvn clean install

Application restart

To apply any changes made, a restart of the application is required. A restart can easily be accomplished by restarting the appropriate container, e.g.:

docker restart shogun-boot

Remote debugger

To create a remote debugger, a new run configuration in IntelliJ has to be created:

  • Open Edit configurations in the Run menu.
  • Add a new Remote JVM debug configuration and enter the following properties:
    • Name: shogun-boot remote debugger
    • Host: localhost
    • Port: 4711 (may be adjusted to the given module/service)
    • Use module classpath: shogun-boot (or any other module)

GeoServer interceptor

To use REST API of GeoServer interceptor it's necessary to create a role interceptor_admin in Keycloak. Users having this role are allowed to use them.

MVN Report

If you want to create a report with detailed information about the projects dependencies or similar, you can use this command:

mvn site -Preporting

Just have a look at /target/site/index.html afterwards.

Get an access token programmatically

To get an access token programmatically the following curl can be used (adjust <EXTERNAL_KEYCLOAK_HOST> beforehand):

curl \
  -v \
  -k \
  -X POST \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d 'grant_type=password' \
  -d 'client_id=shogun-client' \
  -d 'username=shogun' \
  -d 'password=shogun' \
  'https://<EXTERNAL_KEYCLOAK_HOST>/auth/realms/SHOGun/protocol/openid-connect/token' | jq '.access_token'

The usage of jq is optional and can be removed if not needed or jq is not available.

Actuator

Spring Boot Actuator is enabled by default and is available via the actuator/ endpoints. The following list demonstrates some use cases:

  • List current properties:

    • https://localhost/actuator/configprops
  • List current status of flyway migrations:

    • https://localhost/actuator/flyway
  • List current health information:

    • https://localhost/actuator/health
  • List build and git information:

    • https://localhost/actuator/info
  • List current log levels:

    • https://localhost/actuator/loggers
  • List current log level of a specific module:

    • https://localhost/actuator/loggers/de.terrestris
  • Set log level for a specific module to the desired level (e.g. DEBUG for de.terrestris):

    curl \
      -v \
      -X POST \
      -H 'Authorization: Bearer <TOKEN>' \
      -H 'Content-Type: application/json' \
      -d '{"configuredLevel": "DEBUG"}' \
      'https://localhost/actuator/loggers/de.terrestris'
  • List all available endpoint mappings:

    • https://localhost/actuator/mappings

Note: All endpoints are accessible by users with the ADMIN role only.

Release

  • To trigger a release, manually run the Publish github action

Entity Auditing

SHOGun supports auditing of entities, powered by Hibernate Envers.

Auditing is enabled by default and can be disabled by setting spring.jpa.properties.hibernate.integration.envers to false.

Enabling envers mid-project

If envers is enabled mid-way and there is already data this can result in errors when querying audit data. To fix this, a revision with revision type 0 (created) has to be manually inserted for each existing entity into the respective audit table.

See https://discourse.hibernate.org/t/safe-envers-queries-when-the-audit-history-is-incomplete/771.

shogun's People

Contributors

ahennr avatar amandatamanda avatar annarieger avatar buehner avatar dependabot-preview[bot] avatar dependabot[bot] avatar dnlkoch avatar faouzih21 avatar fritzhoing avatar hblitza avatar hwbllmnn avatar kaivolland avatar lukaslohoff avatar marcjansen avatar mholthausen avatar semantic-release-bot avatar simonseyock avatar weskamm avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

shogun's Issues

speed up setPermission methods

I'm trying to create ~20000 new entities and also set permissions for every one. Unfortunately, GroupInstancePermissionService::setPermission takes about 3 seconds per permission.

See

public void setPermission(BaseEntity persistedEntity, PermissionCollectionType permissionCollectionType) {
for example

Ideas:

  • is the check for existing permissions necessary (especially for new entities)? could this be optional maybe?
  • we could provide a method setPermission(List<BaseEntity> persistedEntities, Group group, PermissionCollectionType pct) where repository.saveAll() can be used to create many permissions for a single group

Other suggestions are welcome…

Support for partial updates

We should add interfaces for partial updates of entities.

HTTP PATCH should be appropriate for this case.

Currently if we have large objects or objects with read-only fields we need to send huge objects and do some crazy workarounds.
When changing just one attribute of an entity, there should be an interface to post only this one changed attribute.

The automated release is failing 🚨

🚨 The automated release from the main branch failed. 🚨

I recommend you give this issue a high priority, so other packages depending on you can benefit from your bug fixes and new features again.

You can find below the list of errors reported by semantic-release. Each one of them has to be resolved in order to automatically publish your package. I’m sure you can fix this 💪.

Errors are usually caused by a misconfiguration or an authentication problem. With each error reported below you will find explanation and guidance to help you to resolve it.

Once all the errors are resolved, semantic-release will release your package the next time you push a commit to the main branch. You can also manually restart the failed CI job that runs semantic-release.

If you are not sure how to resolve this, here are some links that can help you:

If those don’t help, or if this issue is reporting something you think isn’t right, you can always ask the humans behind semantic-release.


failed to deploy to maven

Unfortunately this error doesn't have any additional information. Feel free to kindly ask the author of the @terrestris/maven-semantic-release plugin to add more helpful information.


Good luck with your project ✨

Your semantic-release bot 📦🚀

`BaseService.update` returns object that is not a hibernate proxy

The BaseService.update method serializes the given entity, deserializes it again and saves it. This causes the resulting object to not be a proper hibernate proxy. (edit: to be more precise: any related objects are no hibernate proxies anymore, but merely the objects that got deserialized. It can be that the object only contains an Id if the field was using the BaseEntityIdSerializer). Also all other managed instances of the object are broken as well.

Firstly the question arises why this code with the serialization and deserialization exists:

    @PreAuthorize("hasRole('ROLE_ADMIN') or hasPermission(#entity, 'UPDATE')")
    @Transactional(isolation = Isolation.SERIALIZABLE)
    public S update(Long id, S entity) throws IOException {
        Optional<S> persistedEntityOpt = repository.findById(id);

        ObjectNode jsonObject = objectMapper.valueToTree(entity);

        // Ensure the created timestamp will not be overridden.
        S persistedEntity = persistedEntityOpt.orElseThrow();
        OffsetDateTime createdTimestamp = persistedEntity.getCreated();
        String serialized = createdTimestamp != null ? createdTimestamp.toInstant().toString() : null;
        jsonObject.put("created", serialized);

        S updatedEntity = objectMapper.readerForUpdating(persistedEntity).readValue(jsonObject);

        return repository.save(updatedEntity);
    }

I suppose this in place to leave the entity intact. Is this really neccessary?

The simplest way to achieve the same would be:

  entity.setCreated(persistedEntity.getCreated());

Another way to solve this would be to use the lombok @SuperBuilder annotation and create a copy. See https://www.projectlombok.org/features/experimental/SuperBuilder

For our project we used the following workaround:

            var updatedEntity = super.update(entityId, entity);

            entityManager.flush(); // flush database changes
            entityManager.detach(updatedEntity); // this way the broken object is no longer a managed hibernate object
            // without the detach the findById would just return the broken managed object
            updatedEntity = repository.findById(entityId).orElseThrow(); // get a new managed hibernate proxy object

This might not be feasible for the BaseService as I assume that flush does have performance issues if called often.

Modifications of roles for users

The default user shogun is not assigned the role ADMIN by default. Therefore some requests can not be executed (e.g. the actuator). Currently, this must be done manually via Keycloak.

The script for setting up the default users must be modified accordingly.
It is suggested to use the JSON Export/Import function of Keycloak to set up all necessary data.

Handling of flyway migrations

If we introduce new flyway migrations (that should for example be necessary for #247 and #258) this will work for a standalone shogun-boot setup, but project implementations might fail during applying the migrations as they might have added migrations already. As an example: Adding V1.0.0__my_new_shogun_migration.sql would breakup the migration order if the custom project already contains V2.0.0__my_existing_project_migration.sql (V1.0.0 < V2.0.0).

Any ideas on how to get around this problem?

Cleanup duplicated methods of KeycloakUtil and SecurityContextUtil

The util classes KeycloakUtil and SecurityContextUtil contain several more or less identical methods.
This needs to be cleaned up.

e.g:

  • KeycloakUtil.getKeycloakUserIdFromAuthentication vs. SecurityContextUtil.getKeycloakUserIdFromAuthentication
  • KeycloakUtil.getUserGroups vs SecurityContextUtil.getKeycloakGroupsForUser

`mvn package` should not trigger jib build

the jib build is triggered on mvn package. In the pom.xml is this config for the jib plugin:

          <executions>
            <execution>
              <id>dockerBuild</id>
              <goals>
                <goal>dockerBuild</goal>
              </goals>
              <phase>package</phase>
            </execution>
            <execution>
              <id>build</id>
              <goals>
                <goal>build</goal>
              </goals>
              <phase>deploy</phase>
            </execution>
          </executions>

It might be enough to change the phase of dockerBuild to deploy. What do you think?

spring-boot update to v2.4.0 needs adapdations of application.yml files

With #164 the version of spring-boot dependency was updated from 2.3.5.RELEASE to 2.4.0.

With this change the config file processing in Spring Boot was changed (s. https://spring.io/blog/2020/08/14/config-file-processing-in-spring-boot-2-4#multi-document-properties-files) so the existing structure of application.yml files does't match anymore and customized project settings (e.g. profiles) doesn't get overriden in children projects.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.