Skip to content

GitLab

Commit 8ead1e8c authored by Rainer Türner's avatar Rainer Türner
Browse files
Options

Andmemuundur 2.0 initial commit

parent 9f19d7b1
Pipeline #385 failed with stages
      Expand all Hide whitespace changes
      Inline Side-by-side
      Showing with 5459 additions and 1 deletion
      LICENSE.md 0 → 100644
      View file @ 8ead1e8c
      The MIT License
      Copyright (c) 2020 Estonian Information System Authority (RIA)
      Permission is hereby granted, free of charge, to any person obtaining a copy
      of this software and associated documentation files (the "Software"), to deal
      in the Software without restriction, including without limitation the rights
      to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
      copies of the Software, and to permit persons to whom the Software is
      furnished to do so, subject to the following conditions:
      The above copyright notice and this permission notice shall be included in all
      copies or substantial portions of the Software.
      THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
      IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
      FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
      AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
      LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
      OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
      SOFTWARE.
      \ No newline at end of file
      README.md
      View file @ 8ead1e8c
      # Andmemuundur
      JSON mapper + HTML -> PDF formatting
      \ No newline at end of file
      ## About
      Andmemuundur is a common component for data formatting. Its main purpose is to transform backend JSON responses into a format that frontend is capable of printing out without any further data manipulation.
      >**Why is this important?** The less data transformations are made in frontend, the less debugging, polyfilling etc needs to be done to achieve and maintain the same results with different frontend frameworks.
      In addition, the current version of Andmemuundur is also capable of transforming HTML input into PDF and return it in base64 format.
      ## Development and licencing
      | | |
      |-|-|
      | **Licence** | MIT |
      | **Current version** | 2.0 |
      | **Initially developed for** | State Portal eesti.ee, Information System Authority, Republic of Estonia |
      | **Architect** | **Version 1.0** - Rainer Türner, Information System Authority, Republic of Estonia<br> **Version 2.0** - Rainer Türner, Information System Authority, Republic of Estonia |
      | **Developers** | **Version 1.0** - Avalanche Laboratory OÜ / Vladislav Alenitsev<br>**Version 2.0** - Reach-U AS / lead developers Lauri All and Jens-Konrad Preem<br> |
      | **Main usage** | Core backend component of Estonian State Portal eesti.ee since December 2018 |
      | **Maturity** | Versions 1.0 and 2.0 pentested, stable in production |
      | **Plans for the future** | 1. Full coverage of usage by sample configurations;<br>2. Docker images to easily test locally;<br>3. Re-write of core functionality to make service configurations easier to write. |
      pom.xml 0 → 100644
      View file @ 8ead1e8c
      <?xml version="1.0" encoding="UTF-8"?>
      <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
      <modelVersion>4.0.0</modelVersion>
      <groupId>rig.muundur</groupId>
      <artifactId>andmemuundur</artifactId>
      <version>0.0.4-SNAPSHOT</version>
      <packaging>war</packaging>
      <name>andmemuundur</name>
      <description>JSON structure converter for RIG</description>
      <properties>
      <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
      <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
      <java.version>1.8</java.version>
      <maven.compiler.source>1.8</maven.compiler.source>
      <maven.compiler.target>1.8</maven.compiler.target>
      </properties>
      <parent>
      <groupId>ee.ria</groupId>
      <artifactId>pom</artifactId>
      <version>1.0.0-SNAPSHOT</version>
      </parent>
      <dependencyManagement>
      <dependencies>
      <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-parent</artifactId>
      <version>2.0.6.RELEASE</version>
      <type>pom</type>
      <scope>import</scope>
      </dependency>
      </dependencies>
      </dependencyManagement>
      <repositories>
      <repository>
      <id>credentials</id>
      <url>https://example.com/repository/maven-public</url>
      <releases>
      <enabled>true</enabled>
      </releases>
      <snapshots>
      <enabled>true</enabled>
      </snapshots>
      </repository>
      </repositories>
      <dependencies>
      <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-web</artifactId>
      </dependency>
      <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-aop</artifactId>
      </dependency>
      <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-webflux</artifactId>
      </dependency>
      <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-devtools</artifactId>
      <scope>runtime</scope>
      </dependency>
      <dependency>
      <groupId>org.projectlombok</groupId>
      <artifactId>lombok</artifactId>
      <optional>true</optional>
      </dependency>
      <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-test</artifactId>
      <scope>test</scope>
      </dependency>
      <dependency>
      <groupId>io.projectreactor</groupId>
      <artifactId>reactor-test</artifactId>
      <scope>test</scope>
      </dependency>
      <dependency>
      <groupId>io.springfox</groupId>
      <artifactId>springfox-swagger2</artifactId>
      <version>2.9.2</version>
      </dependency>
      <dependency>
      <groupId>io.springfox</groupId>
      <artifactId>springfox-swagger-ui</artifactId>
      <version>2.9.2</version>
      </dependency>
      <dependency>
      <groupId>com.fasterxml.jackson.core</groupId>
      <artifactId>jackson-databind</artifactId>
      <version>2.9.4</version>
      <scope>compile</scope>
      </dependency>
      <dependency>
      <groupId>com.github.jknack</groupId>
      <artifactId>handlebars</artifactId>
      <version>4.0.6</version>
      <scope>compile</scope>
      </dependency>
      <dependency>
      <groupId>net.sf.jtidy</groupId>
      <artifactId>jtidy</artifactId>
      <version>r938</version>
      <scope>compile</scope>
      </dependency>
      <dependency>
      <groupId>org.xhtmlrenderer</groupId>
      <artifactId>flying-saucer-pdf-itext5</artifactId>
      <version>9.1.6</version>
      <scope>compile</scope>
      </dependency>
      <dependency>
      <groupId>org.apache.pdfbox</groupId>
      <artifactId>pdfbox</artifactId>
      <version>2.0.6</version>
      </dependency>
      <dependency>
      <groupId>ee.ria.commons</groupId>
      <artifactId>id-log</artifactId>
      <version>1.0.0-SNAPSHOT</version>
      </dependency>
      </dependencies>
      <build>
      <finalName>${project.artifactId}</finalName>
      <plugins>
      <plugin>
      <artifactId>maven-war-plugin</artifactId>
      <configuration>
      <failOnMissingWebXml>false</failOnMissingWebXml>
      </configuration>
      </plugin>
      <plugin>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-maven-plugin</artifactId>
      </plugin>
      <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-surefire-plugin</artifactId>
      <version>2.22.1</version>
      <configuration>
      <useSystemClassLoader>false</useSystemClassLoader>
      </configuration>
      </plugin>
      </plugins>
      </build>
      </project>
      readme/AM_INSTALLATION_GUIDE.md 0 → 100644
      View file @ 8ead1e8c
      # Dependencies
      Java 11, Maven, Tomcat 9
      Tested on Ubuntu 18.04.4 LTS.
      # Install dependencies
      ## Install Java
      ```
      sudo apt install openjdk-11-jdk
      ```
      ## Install Maven
      ```
      sudo apt install Maven
      ```
      ### Setup Tomcat user
      For security purposes, Tomcat should not be run under the root user. In this guide we create a new system user and group with home directory /opt/Tomcat that will run the Tomcat service.
      ```
      sudo useradd -r -m -U -d /opt/Tomcat -s /bin/false Tomcat
      ```
      ### Install Tomcat 9
      Download latest binary from the official Tomcat project page
      https://Tomcat.apache.org/download-90.cgi
      For example with wget
      ```
      wget https://downloads.apache.org/Tomcat/Tomcat-9/v9.0.31/bin/apache-Tomcat-9.0.31.tar.gz -P /tmp
      ```
      At the time of writing this instruction, the latest Tomcat 9 is 9.0.31.
      I replace the minor versions with * in further instructions, as this might be reliable to change.
      Extract the downloaded archive
      ```
      sudo tar xf /tmp/apache-Tomcat-9*.tar.gz -C /opt/Tomcat
      ```
      Create symlink for ease of use
      ```
      sudo ln -s /opt/Tomcat/apache-Tomcat-9.*.* /opt/Tomcat/latest
      ```
      Change the owner of installation folder to Tomcat user and make scripts executable
      ```
      sudo chown -RH Tomcat: /opt/Tomcat/latest
      sudo sh -c 'chmod +x /opt/Tomcat/latest/bin/*.sh'
      ```
      ### Optional: create Tomcat service
      Though you can start and stop Tomcat manually
      ```
      opt/Tomcat/latest/bin/startup.sh
      /opt/Tomcat/latest/bin/shutdown.sh
      ```
      For comfortable use Tomcat should be set up as a service.
      Using your favorite text editor create a file
      ```
      /etc/systemd/system/Tomcat.service
      ```
      with content ...
      ```
      [Unit]
      Description=Tomcat 9 servlet container
      After=network.target
      [Service]
      Type=forking
      User=Tomcat
      Group=Tomcat
      Environment="JAVA_HOME=/usr/lib/jvm/default-java"
      Environment="JAVA_OPTS=-Djava.security.egd=file:///dev/urandom -Djava.awt.headless=true"
      Environment="CATALINA_BASE=/opt/Tomcat/latest"
      Environment="CATALINA_HOME=/opt/Tomcat/latest"
      Environment="CATALINA_PID=/opt/Tomcat/latest/temp/Tomcat.pid"
      Environment="CATALINA_OPTS=-Xms512M -Xmx1024M -server -XX:+UseParallelGC"
      ExecStart=/opt/Tomcat/latest/bin/startup.sh
      ExecStop=/opt/Tomcat/latest/bin/shutdown.sh
      [Install]
      WantedBy=multi-user.target
      check your Java home -- in my installation the Java installation produced
      /usr/lib/jvm/java-1.11.0-openjd-amd64
      ```
      Symlink pointing to Java installation directory
      ```
      /usr/lib/jvmjava-11-openjdk-amd64
      ```
      One could paste that into the above file. I chose to rename the symlink.
      ```
      sudo mv /usr/lib/jvm/java-1-11-0-openjd-amd64 /usr/lib/jvm/java/default-java
      ```
      Reload systemd deamon
      ```
      sudo systemctl daemon-reload
      ```
      Now one can start/stop/restart the service and look its status
      ```
      sudo systemctl start/stop/restart/status Tomcat
      ```
      If status shows Tomcat running nicely one can now enable Tomcat to start on system boot time
      ```
      sudo sustemctl enable Tomcat
      ```
      # Package and install the application
      After downloading the source code.
      Move to the uppermost project folder containing pom.xml file.
      Package the application
      ```
      mvn clean -U package
      ```
      In folder `/target` `andmemuundur.war` package is generated.
      Copy `andmemuundur.war` to `/opt/Tomcat/latest/webapps`. Copy there also `/templates` folder from source code.
      Start or restart Tomcat 9.
      \ No newline at end of file
      readme/README.md 0 → 100644
      View file @ 8ead1e8c
      # Andmemuundur README
      Most of the current functionalities are related to JSON re-mapping/re-formatting
      and are demonstrated in SAMPLE_REQUESTS.md file.
      Most of the sample requests there use existing templates from templates folder.
      For a few others extra example templates are located in templates_examples folder.
      For extra information one can also refer to
      src/test/java/rig/andmemuundur/service/JsonServiceTests.java which contains
      tests that also demonstrate these functionalities. Input output and templates for mapping between I/O for these
      test cases are in the src/test/java/rig/andmemuundur/resources/json subfolders.
      Most of the basic functionality for JSON mapping/conversions can be described by following template :
      ```
      {
      "this_will_be_field_name_in_output":{
      "$.input" :"$.this_is_field_name_in_input",
      }
      }
      ```
      For mapping arrays some more complicated constructions are available to be used in templates :
      ```
      {
      "array_name_in_output" : [
      {
      "#.input_array": "$.array_name_in_input",
      "sort_property": "$.field_from_object_in_that_array",
      "sort_type": "date",
      "sort_date_format": "yyyy-MM-dd'T'HH:mm'Z'",
      "sort_order": "desc",
      "properties": {
      "field_name_in_array_object_in_output": "$.some_field_from_object_in_input_array",
      }
      ]
      }
      ```
      Instead of "#.input_array" one might also see "from_property" field in templates. This should be considered
      a depreciated feature. For writing future templates "#.input_array" is strongly recommended as this would
      allow the same functionality and more.
      For more detailed examples showing more complicated structures SAMPLE_REQUEST.md and its used templates
      should be consulted. Also there are examples for data conversion functionalities at the end of this file.
      Functionality of endpoint json/v1/convert that is also demonstrated in
      SAMPLE_REQUESTS.md does right now not use the general template based configuration.
      Currently only conversion of HTML-string to (base64 encoded) PDF is implemented.
      Fore extra information about this one can also refer to related tests
      src/test/java/rig/andmemuundur/service/PdfServiceTests.java
      Internal functionalities examples.Some of which are also covered in the JSON transformations example
      in SAMPLE_REQUESTS.md.
      Transform date format.
      ```
      {
      "string_to_date": {
      "$.input": "$.aeg",
      "input_format": "dd-MM-yyyy HH:mm:ss.SSS",
      "type": "date",
      "format": "dd.MM.yyyy"
      }
      }
      ```
      Integer json to string json and vice versa
      ```
      {
      "int_to_string": {
      "$.input": "$.int_to_string",
      "type": "string"
      },
      "string_to_int": {
      "$.input": "$.string_to_int",
      "type": "integer"
      }
      }
      ```
      String json to decimal json, decimal json with scale,
      Decimal json to string json or integer json
      ```
      {
      "str_to_decimal": {
      "$.input": "$.str_to_decimal",
      "type": "decimal"
      },
      "str_to_decimal_scale": {
      "$.input": "$.str_to_decimal",
      "type": "decimal",
      "scale": 2
      },
      "decimal_to_str": {
      "$.input": "$.decimal_to_str",
      "type": "string"
      },
      "decimal_to_integer": {
      "$.input": "$.decimal_to_integer",
      "type": "integer"
      }
      }
      ```
      String json to boolean and vice versa
      ```
      {
      "str_to_bln": {
      "$.input": "$.str_to_bln",
      "type": "boolean"
      },
      "bln_to_str": {
      "$.input": "$.bln_to_str",
      "type": "string"
      }
      }
      ```
      \ No newline at end of file
      readme/SAMPLE_REQUESTS.md 0 → 100644
      View file @ 8ead1e8c
      ### 1. Display output as an array
      * Mapping:
      ```templates/aar/aar_oigused.json```
      Array is controlled by the "output_as_array" parameter in the mapping file. In this case "registrikood" is output as array (if more than 1 value exists in response).
      * Request:
      ```
      curl -X POST \
      http://10.0.24.148:8080/andmemuundur/json/v1/aar_oigused \
      -H 'Content-Type: application/json' \
      -H 'cache-control: no-cache' \
      -d '{
      "data": {
      "response": {
      "oigused": {
      "oigus": [
      {
      "oigusNimi": "GENERAL.ARENDUS",
      "registrikood": "70006317"
      },
      {
      "registrikood": "70006318",
      "oigusNimi": "GENERAL.ADMIN"
      }
      ]
      }
      },
      "error": null
      }
      }'
      ```
      output
      ```
      {
      "response": {
      "registrikood": ["70006317", "70006318"]
      }
      }
      ```
      ### 2. Filter output values
      * Mapping: ```templates/rr/RR_44.json```
      Only results containing values "LAPS" and "REGISTRIS" will be shown.
      * Request:
      ```
      curl -X POST \
      http://10.0.24.148:8080/andmemuundur/json/v1/RR_44 \
      -H 'Content-Type: application/json' \
      -H 'cache-control: no-cache' \
      -d '{
      "data": {
      "response": {
      "Suhted": {
      "Suhteandmed": [
      {
      "Suhteandmed.ssuhtetyyp": "LAPS",
      "Suhteandmed.sIsikukood": "1234",
      "Suhteandmed.sStaatus": "REGISTRIS"
      },
      {
      "Suhteandmed.ssuhtetyyp": "VEND",
      "Suhteandmed.sIsikukood": "12345",
      "Suhteandmed.sStaatus": "REGISTRIS"
      },
      {
      "Suhteandmed.ssuhtetyyp": "LAPS",
      "Suhteandmed.sIsikukood": "123456",
      "Suhteandmed.sStaatus": "ARHIIVIS"
      }
      ]
      }
      }
      }
      }'
      ```
      output
      ```
      {
      "response": {
      "i18n_minu_lapsed": [{
      "i18n_suhtetyyp": "LAPS",
      "i18n_isikukood": "1234",
      "i18n_staatus": "REGISTRIS"
      }]
      }
      }
      ```
      ### 3. JSON transformations
      * Mapping: ```templates_examples/transformType.json```
      To test this example the sample template `transformType.json` from templates_examples/ directory should be placed in the templates directory and the service should be restarted.
      * Request:
      ```
      curl -X POST \
      http://10.0.24.148:8080/andmemuundur/json/v1/transformType \
      -H 'Content-Type: application/json' \
      -H 'cache-control: no-cache' \
      -d '{
      "data": {
      "aeg": "12-02-2018 01:34:00.000",
      "int_to_string": 123,
      "string_to_int": "123",
      "str_to_decimal": "123.1234",
      "decimal_to_str": 123.1234,
      "decimal_to_integer": 123.5234,
      "str_to_bln": "true",
      "bln_to_str": false
      }
      }
      '
      ```
      output
      ```
      {
      "string_to_date": "12.02.2018",
      "int_to_string": "123",
      "string_to_int": 123,
      "str_to_decimal": 123.1234,
      "str_to_decimal_scale": 123.12,
      "decimal_to_str": "123.1234",
      "decimal_to_integer": 124,
      "str_to_bln": true,
      "bln_to_str": "false"
      }
      ```
      ### 4. String replacement
      * Mapping: ```templates/inbox/inbox.json```
      Notice the replaced email addresses and the removed `<img>` tag in the response.
      * Request: