Skip to content

GitLab

Commit 8ead1e8c authored by Rainer Türner's avatar Rainer Türner
Browse files

Andmemuundur 2.0 initial commit

parent 9f19d7b1
Pipeline #385 failed with stages
Hide whitespace changes
Inline Side-by-side
Showing with 1235 additions and 1 deletion
LICENSE.md 0 → 100644
View file @ 8ead1e8c
The MIT License
Copyright (c) 2020 Estonian Information System Authority (RIA)
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
\ No newline at end of file
README.md
View file @ 8ead1e8c
# Andmemuundur
JSON mapper + HTML -> PDF formatting
\ No newline at end of file
## About
Andmemuundur is a common component for data formatting. Its main purpose is to transform backend JSON responses into a format that frontend is capable of printing out without any further data manipulation.
>**Why is this important?** The less data transformations are made in frontend, the less debugging, polyfilling etc needs to be done to achieve and maintain the same results with different frontend frameworks.
In addition, the current version of Andmemuundur is also capable of transforming HTML input into PDF and return it in base64 format.
## Development and licencing
| | |
|-|-|
| **Licence** | MIT |
| **Current version** | 2.0 |
| **Initially developed for** | State Portal eesti.ee, Information System Authority, Republic of Estonia |
| **Architect** | **Version 1.0** - Rainer Türner, Information System Authority, Republic of Estonia<br> **Version 2.0** - Rainer Türner, Information System Authority, Republic of Estonia |
| **Developers** | **Version 1.0** - Avalanche Laboratory OÜ / Vladislav Alenitsev<br>**Version 2.0** - Reach-U AS / lead developers Lauri All and Jens-Konrad Preem<br> |
| **Main usage** | Core backend component of Estonian State Portal eesti.ee since December 2018 |
| **Maturity** | Versions 1.0 and 2.0 pentested, stable in production |
| **Plans for the future** | 1. Full coverage of usage by sample configurations;<br>2. Docker images to easily test locally;<br>3. Re-write of core functionality to make service configurations easier to write. |
pom.xml 0 → 100644
View file @ 8ead1e8c
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>rig.muundur</groupId>
<artifactId>andmemuundur</artifactId>
<version>0.0.4-SNAPSHOT</version>
<packaging>war</packaging>
<name>andmemuundur</name>
<description>JSON structure converter for RIG</description>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<java.version>1.8</java.version>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
</properties>
<parent>
<groupId>ee.ria</groupId>
<artifactId>pom</artifactId>
<version>1.0.0-SNAPSHOT</version>
</parent>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.0.6.RELEASE</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<repositories>
<repository>
<id>credentials</id>
<url>https://example.com/repository/maven-public</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-aop</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-webflux</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-devtools</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.projectreactor</groupId>
<artifactId>reactor-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.springfox</groupId>
<artifactId>springfox-swagger2</artifactId>
<version>2.9.2</version>
</dependency>
<dependency>
<groupId>io.springfox</groupId>
<artifactId>springfox-swagger-ui</artifactId>
<version>2.9.2</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.9.4</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>com.github.jknack</groupId>
<artifactId>handlebars</artifactId>
<version>4.0.6</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>net.sf.jtidy</groupId>
<artifactId>jtidy</artifactId>
<version>r938</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.xhtmlrenderer</groupId>
<artifactId>flying-saucer-pdf-itext5</artifactId>
<version>9.1.6</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.pdfbox</groupId>
<artifactId>pdfbox</artifactId>
<version>2.0.6</version>
</dependency>
<dependency>
<groupId>ee.ria.commons</groupId>
<artifactId>id-log</artifactId>
<version>1.0.0-SNAPSHOT</version>
</dependency>
</dependencies>
<build>
<finalName>${project.artifactId}</finalName>
<plugins>
<plugin>
<artifactId>maven-war-plugin</artifactId>
<configuration>
<failOnMissingWebXml>false</failOnMissingWebXml>
</configuration>
</plugin>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.22.1</version>
<configuration>
<useSystemClassLoader>false</useSystemClassLoader>
</configuration>
</plugin>
</plugins>
</build>
</project>
readme/AM_INSTALLATION_GUIDE.md 0 → 100644
View file @ 8ead1e8c
# Dependencies
Java 11, Maven, Tomcat 9
Tested on Ubuntu 18.04.4 LTS.
# Install dependencies
## Install Java
```
sudo apt install openjdk-11-jdk
```
## Install Maven
```
sudo apt install Maven
```
### Setup Tomcat user
For security purposes, Tomcat should not be run under the root user. In this guide we create a new system user and group with home directory /opt/Tomcat that will run the Tomcat service.
```
sudo useradd -r -m -U -d /opt/Tomcat -s /bin/false Tomcat
```
### Install Tomcat 9
Download latest binary from the official Tomcat project page
https://Tomcat.apache.org/download-90.cgi
For example with wget
```
wget https://downloads.apache.org/Tomcat/Tomcat-9/v9.0.31/bin/apache-Tomcat-9.0.31.tar.gz -P /tmp
```
At the time of writing this instruction, the latest Tomcat 9 is 9.0.31.
I replace the minor versions with * in further instructions, as this might be reliable to change.
Extract the downloaded archive
```
sudo tar xf /tmp/apache-Tomcat-9*.tar.gz -C /opt/Tomcat
```
Create symlink for ease of use
```
sudo ln -s /opt/Tomcat/apache-Tomcat-9.*.* /opt/Tomcat/latest
```
Change the owner of installation folder to Tomcat user and make scripts executable
```
sudo chown -RH Tomcat: /opt/Tomcat/latest
sudo sh -c 'chmod +x /opt/Tomcat/latest/bin/*.sh'
```
### Optional: create Tomcat service
Though you can start and stop Tomcat manually
```
opt/Tomcat/latest/bin/startup.sh
/opt/Tomcat/latest/bin/shutdown.sh
```
For comfortable use Tomcat should be set up as a service.
Using your favorite text editor create a file
```
/etc/systemd/system/Tomcat.service
```
with content ...
```
[Unit]
Description=Tomcat 9 servlet container
After=network.target
[Service]
Type=forking
User=Tomcat
Group=Tomcat
Environment="JAVA_HOME=/usr/lib/jvm/default-java"
Environment="JAVA_OPTS=-Djava.security.egd=file:///dev/urandom -Djava.awt.headless=true"
Environment="CATALINA_BASE=/opt/Tomcat/latest"
Environment="CATALINA_HOME=/opt/Tomcat/latest"
Environment="CATALINA_PID=/opt/Tomcat/latest/temp/Tomcat.pid"
Environment="CATALINA_OPTS=-Xms512M -Xmx1024M -server -XX:+UseParallelGC"
ExecStart=/opt/Tomcat/latest/bin/startup.sh
ExecStop=/opt/Tomcat/latest/bin/shutdown.sh
[Install]
WantedBy=multi-user.target
check your Java home -- in my installation the Java installation produced
/usr/lib/jvm/java-1.11.0-openjd-amd64
```
Symlink pointing to Java installation directory
```
/usr/lib/jvmjava-11-openjdk-amd64
```
One could paste that into the above file. I chose to rename the symlink.
```
sudo mv /usr/lib/jvm/java-1-11-0-openjd-amd64 /usr/lib/jvm/java/default-java
```
Reload systemd deamon
```
sudo systemctl daemon-reload
```
Now one can start/stop/restart the service and look its status
```
sudo systemctl start/stop/restart/status Tomcat
```
If status shows Tomcat running nicely one can now enable Tomcat to start on system boot time
```
sudo sustemctl enable Tomcat
```
# Package and install the application
After downloading the source code.
Move to the uppermost project folder containing pom.xml file.
Package the application
```
mvn clean -U package
```
In folder `/target` `andmemuundur.war` package is generated.
Copy `andmemuundur.war` to `/opt/Tomcat/latest/webapps`. Copy there also `/templates` folder from source code.
Start or restart Tomcat 9.
\ No newline at end of file
readme/README.md 0 → 100644
View file @ 8ead1e8c
# Andmemuundur README
Most of the current functionalities are related to JSON re-mapping/re-formatting
and are demonstrated in SAMPLE_REQUESTS.md file.
Most of the sample requests there use existing templates from templates folder.
For a few others extra example templates are located in templates_examples folder.
For extra information one can also refer to
src/test/java/rig/andmemuundur/service/JsonServiceTests.java which contains
tests that also demonstrate these functionalities. Input output and templates for mapping between I/O for these
test cases are in the src/test/java/rig/andmemuundur/resources/json subfolders.
Most of the basic functionality for JSON mapping/conversions can be described by following template :
```
{
"this_will_be_field_name_in_output":{
"$.input" :"$.this_is_field_name_in_input",
}
}
```
For mapping arrays some more complicated constructions are available to be used in templates :
```
{
"array_name_in_output" : [
{
"#.input_array": "$.array_name_in_input",
"sort_property": "$.field_from_object_in_that_array",
"sort_type": "date",
"sort_date_format": "yyyy-MM-dd'T'HH:mm'Z'",
"sort_order": "desc",
"properties": {
"field_name_in_array_object_in_output": "$.some_field_from_object_in_input_array",
}
]
}
```
Instead of "#.input_array" one might also see "from_property" field in templates. This should be considered
a depreciated feature. For writing future templates "#.input_array" is strongly recommended as this would
allow the same functionality and more.
For more detailed examples showing more complicated structures SAMPLE_REQUEST.md and its used templates
should be consulted. Also there are examples for data conversion functionalities at the end of this file.
Functionality of endpoint json/v1/convert that is also demonstrated in
SAMPLE_REQUESTS.md does right now not use the general template based configuration.
Currently only conversion of HTML-string to (base64 encoded) PDF is implemented.
Fore extra information about this one can also refer to related tests
src/test/java/rig/andmemuundur/service/PdfServiceTests.java
Internal functionalities examples.Some of which are also covered in the JSON transformations example
in SAMPLE_REQUESTS.md.
Transform date format.
```
{
"string_to_date": {
"$.input": "$.aeg",
"input_format": "dd-MM-yyyy HH:mm:ss.SSS",
"type": "date",
"format": "dd.MM.yyyy"
}
}
```
Integer json to string json and vice versa
```
{
"int_to_string": {
"$.input": "$.int_to_string",
"type": "string"
},
"string_to_int": {
"$.input": "$.string_to_int",
"type": "integer"
}
}
```
String json to decimal json, decimal json with scale,
Decimal json to string json or integer json
```
{
"str_to_decimal": {
"$.input": "$.str_to_decimal",
"type": "decimal"
},
"str_to_decimal_scale": {
"$.input": "$.str_to_decimal",
"type": "decimal",
"scale": 2
},
"decimal_to_str": {
"$.input": "$.decimal_to_str",
"type": "string"
},
"decimal_to_integer": {
"$.input": "$.decimal_to_integer",
"type": "integer"
}
}
```
String json to boolean and vice versa
```
{
"str_to_bln": {
"$.input": "$.str_to_bln",
"type": "boolean"
},
"bln_to_str": {
"$.input": "$.bln_to_str",
"type": "string"
}
}
```
\ No newline at end of file
readme/SAMPLE_REQUESTS.md 0 → 100644
View file @ 8ead1e8c
### 1. Display output as an array
* Mapping:
```templates/aar/aar_oigused.json```
Array is controlled by the "output_as_array" parameter in the mapping file. In this case "registrikood" is output as array (if more than 1 value exists in response).
* Request:
```
curl -X POST \
http://10.0.24.148:8080/andmemuundur/json/v1/aar_oigused \
-H 'Content-Type: application/json' \
-H 'cache-control: no-cache' \
-d '{
"data": {
"response": {
"oigused": {
"oigus": [
{
"oigusNimi": "GENERAL.ARENDUS",
"registrikood": "70006317"
},
{
"registrikood": "70006318",
"oigusNimi": "GENERAL.ADMIN"
}
]
}
},
"error": null
}
}'
```
output
```
{
"response": {
"registrikood": ["70006317", "70006318"]
}
}
```
### 2. Filter output values
* Mapping: ```templates/rr/RR_44.json```
Only results containing values "LAPS" and "REGISTRIS" will be shown.
* Request:
```
curl -X POST \
http://10.0.24.148:8080/andmemuundur/json/v1/RR_44 \
-H 'Content-Type: application/json' \
-H 'cache-control: no-cache' \
-d '{
"data": {
"response": {
"Suhted": {
"Suhteandmed": [
{
"Suhteandmed.ssuhtetyyp": "LAPS",
"Suhteandmed.sIsikukood": "1234",
"Suhteandmed.sStaatus": "REGISTRIS"
},
{
"Suhteandmed.ssuhtetyyp": "VEND",
"Suhteandmed.sIsikukood": "12345",
"Suhteandmed.sStaatus": "REGISTRIS"
},
{
"Suhteandmed.ssuhtetyyp": "LAPS",
"Suhteandmed.sIsikukood": "123456",
"Suhteandmed.sStaatus": "ARHIIVIS"
}
]
}
}
}
}'
```
output
```
{
"response": {
"i18n_minu_lapsed": [{
"i18n_suhtetyyp": "LAPS",
"i18n_isikukood": "1234",
"i18n_staatus": "REGISTRIS"
}]
}
}
```
### 3. JSON transformations
* Mapping: ```templates_examples/transformType.json```
To test this example the sample template `transformType.json` from templates_examples/ directory should be placed in the templates directory and the service should be restarted.
* Request:
```
curl -X POST \
http://10.0.24.148:8080/andmemuundur/json/v1/transformType \
-H 'Content-Type: application/json' \
-H 'cache-control: no-cache' \
-d '{
"data": {
"aeg": "12-02-2018 01:34:00.000",
"int_to_string": 123,
"string_to_int": "123",
"str_to_decimal": "123.1234",
"decimal_to_str": 123.1234,
"decimal_to_integer": 123.5234,
"str_to_bln": "true",
"bln_to_str": false
}
}
'
```
output
```
{
"string_to_date": "12.02.2018",
"int_to_string": "123",
"string_to_int": 123,
"str_to_decimal": 123.1234,
"str_to_decimal_scale": 123.12,
"decimal_to_str": "123.1234",
"decimal_to_integer": 124,
"str_to_bln": true,
"bln_to_str": "false"
}
```
### 4. String replacement
* Mapping: ```templates/inbox/inbox.json```
Notice the replaced email addresses and the removed `<img>` tag in the response.
* Request:
```
curl -X POST \
http://10.0.24.148:8080/andmemuundur/json/v1/inbox \
-H 'Content-Type: application/json' \
-H 'cache-control: no-cache' \
-d '{
"data": {
"summary": "Summary of the message",
"messages": [
{
"from": {
"sent_from_email": "email@inbox.www.eesti.ee",
"sent_from_name": "Tester Dude"
},
"content": {
"subject": "Subject",
"content": {
"ISO-8859-1": {