Commit f543883b authored by Rainer Türner's avatar Rainer Türner
Browse files

MVP v1 developments by Net Group

parent c8cd1247
Pipeline #3771 failed with stages
in 0 seconds
#Dependencies :
java 11, Maven, tomcat9
Tested on Ubuntu 18.04.4 LTS.
#Install dependencies.
##Install java
sudo apt install openjdk-11-jdk
##Install maven
sudo apt install maven
###Setup tomcat user
For security purposes, Tomcat should not be run under the root user. In this guide we create a new system user and group with home directory /opt/tomcat that will run the Tomcat service.
sudo useradd -r -m -U -d /opt/tomcat -s /bin/false tomcat
###install tomcat9
download latest binary from the official tomcat project page
https://tomcat.apache.org/download-90.cgi
for example with wget
wget https://downloads.apache.org/tomcat/tomcat-9/v9.0.31/bin/apache-tomcat-9.0.31.tar.gz -P /tmp
At the time of writing this instruction
the latest tomcat9 is 9.0.31.
I replace the minor versions with * in further instructions, as this might be reliable to change.
Extract the downloaded archive
sudo tar xf /tmp/apache-tomcat-9*.tar.gz -C /opt/tomcat
Create symlink for ease of use
sudo ln -s /opt/tomcat/apache-tomcat-9.*.* /opt/tomcat/latest
Change the owner of installation folder to tomcat user and make scripts executable
sudo chown -RH tomcat: /opt/tomcat/latest
sudo sh -c 'chmod +x /opt/tomcat/latest/bin/*.sh'
###optional : create tomcat service
Though you can start and stop tomcat manually
opt/tomcat/latest/bin/startup.sh
/opt/tomcat/latest/bin/shutdown.sh
For comfortable use tomcat should be set up as a service.
Using your favorite text editor create a file
/etc/systemd/system/tomcat.service
with content ...
[Unit]
Description=Tomcat 9 servlet container
After=network.target
[Service]
Type=forking
User=tomcat
Group=tomcat
Environment="JAVA_HOME=/usr/lib/jvm/default-java"
Environment="JAVA_OPTS=-Djava.security.egd=file:///dev/urandom -Djava.awt.headless=true"
Environment="CATALINA_BASE=/opt/tomcat/latest"
Environment="CATALINA_HOME=/opt/tomcat/latest"
Environment="CATALINA_PID=/opt/tomcat/latest/temp/tomcat.pid"
Environment="CATALINA_OPTS=-Xms512M -Xmx1024M -server -XX:+UseParallelGC"
ExecStart=/opt/tomcat/latest/bin/startup.sh
ExecStop=/opt/tomcat/latest/bin/shutdown.sh
[Install]
WantedBy=multi-user.target
check your java home -- in my installation the java installation produced
/usr/lib/jvm/java-1.11.0-openjd-amd64
symlink pointing to java installation directory
/usr/lib/jvmjava-11-openjdk-amd64
one could paste that into the above file
I chose to rename symlink
sudo mv /usr/lib/jvm/java-1-11-0-openjd-amd64 /usr/lib/jvm/java/default-java
reload systemd deamon
sudo systemctl daemon-reload
now one can start/stop/restart the service and look its status
sudo systemctl start/stop/restart/status tomcat
if status shows tomcat running nicely one can now enable tomcat to start on system boot time
sudo sustemctl enable tomcat
#Package and install the application
After downloading the source code.
Move to the uppermost project folder containing pom.xml file.
Package the application
mvn clean -U package
in folder /target andmemuundur.war package is generated.
Copy andmemuundur.war to /opt/tomcat/latest/webapps. Copy there also /templates folder from source code.
Start or restart tomcat9.
MIT License
Copyright (c) 2022 buerokratt
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
\ No newline at end of file
# DMapper
#####/v1/{code} endpoint
Most of the current functionalities are related to JSON re-mapping/re-formatting
and are demonstrated in SAMPLE_REQUESTS.md file.
Most of the sample requests there use existing templates from templates folder.
For a few others extra example templates are located in template_examples folder.
For extra information one can also refer to
src/test/java/rig/andmemuundur/service/JsonServiceTests.java which contains
tests that also demonstrate these functionalities. Input, output and templates for mapping between I/O for these
test cases are in the src/test/java/rig/andmemuundur/resources/json subfolders.
## Getting started
Mapping files are located at templates/v1 directory.
To make it easy for you to get started with GitLab, here's a list of recommended next steps.
Already a pro? Just edit this README.md and make it your own. Want to make it easy? [Use the template at the bottom](#editing-this-readme)!
## Add your files
Most of the basic functionality for JSON mapping/conversions can be described by following template :
```
{
"this_will_be_field_name_in_output":{
"$.input" :"$.this_is_field_name_in_input",
}
}
- [ ] [Create](https://docs.gitlab.com/ee/user/project/repository/web_editor.html#create-a-file) or [upload](https://docs.gitlab.com/ee/user/project/repository/web_editor.html#upload-a-file) files
- [ ] [Add files using the command line](https://docs.gitlab.com/ee/gitlab-basics/add-file.html#add-a-file-using-the-command-line) or push an existing Git repository with the following command:
```
For mapping arrays some more complicated constructions are available to be used in templates :
```
cd existing_repo
git remote add origin https://koodivaramu.eesti.ee/buerokratt/plug-and-play/dmapper.git
git branch -M main
git push -uf origin main
{
"array_name_in_output" : [
{
"#.input_array": "$.array_name_in_input",
"sort_property": "$.field_from_object_in_that_array",
"sort_type": "date",
"sort_date_format": "yyyy-MM-dd'T'HH:mm'Z'",
"sort_order": "desc",
"properties": {
"field_name_in_array_object_in_output": "$.some_field_from_object_in_input_array",
}
]
}
```
Instead of "#.input_array" one might also see "from_property" field in templates. This should be considered
a depreciated feature. For writing future templates "#.input_array" is strongly recommended as this would
allow the same functionality and more.
For more detailed examples showing more complicated structures SAMPLE_REQUEST.md and its used templates
should be consulted. Also there are examples for data conversion functionalities at the end of this file.
## Integrate with your tools
- [ ] [Set up project integrations](https://koodivaramu.eesti.ee/buerokratt/plug-and-play/dmapper/-/settings/integrations)
## Collaborate with your team
- [ ] [Invite team members and collaborators](https://docs.gitlab.com/ee/user/project/members/)
- [ ] [Create a new merge request](https://docs.gitlab.com/ee/user/project/merge_requests/creating_merge_requests.html)
- [ ] [Automatically close issues from merge requests](https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically)
- [ ] [Enable merge request approvals](https://docs.gitlab.com/ee/user/project/merge_requests/approvals/)
- [ ] [Automatically merge when pipeline succeeds](https://docs.gitlab.com/ee/user/project/merge_requests/merge_when_pipeline_succeeds.html)
## Test and Deploy
Use the built-in continuous integration in GitLab.
- [ ] [Get started with GitLab CI/CD](https://docs.gitlab.com/ee/ci/quick_start/index.html)
- [ ] [Analyze your code for known vulnerabilities with Static Application Security Testing(SAST)](https://docs.gitlab.com/ee/user/application_security/sast/)
- [ ] [Deploy to Kubernetes, Amazon EC2, or Amazon ECS using Auto Deploy](https://docs.gitlab.com/ee/topics/autodevops/requirements.html)
- [ ] [Use pull-based deployments for improved Kubernetes management](https://docs.gitlab.com/ee/user/clusters/agent/)
- [ ] [Set up protected environments](https://docs.gitlab.com/ee/ci/environments/protected_environments.html)
***
Functionality of endpoint json/v1/convert that is also demonstrated in
SAMPLE_REQUESTS.md does right now not use the general template based configuration.
Currently only conversion of HTML-string to (base64 encoded) PDF is implemented.
# Editing this README
Fore extra information about this one can also refer to related tests
src/test/java/rig/andmemuundur/service/PdfServiceTests.java
When you're ready to make this README your own, just edit this file and use the handy template below (or feel free to structure it however you want - this is just a starting point!). Thank you to [makeareadme.com](https://www.makeareadme.com/) for this template.
Internal functionalities examples.Some of which are also covered in the JSON transformations example
in SAMPLE_REQUESTS.md.
## Suggestions for a good README
Every project is different, so consider which of these sections apply to yours. The sections used in the template are suggestions for most open source projects. Also keep in mind that while a README can be too long and detailed, too long is better than too short. If you think your README is too long, consider utilizing another form of documentation rather than cutting out information.
## Name
Choose a self-explaining name for your project.
## Description
Let people know what your project can do specifically. Provide context and add a link to any reference visitors might be unfamiliar with. A list of Features or a Background subsection can also be added here. If there are alternatives to your project, this is a good place to list differentiating factors.
## Badges
On some READMEs, you may see small images that convey metadata, such as whether or not all the tests are passing for the project. You can use Shields to add some to your README. Many services also have instructions for adding a badge.
## Visuals
Depending on what you are making, it can be a good idea to include screenshots or even a video (you'll frequently see GIFs rather than actual videos). Tools like ttygif can help, but check out Asciinema for a more sophisticated method.
## Installation
Within a particular ecosystem, there may be a common way of installing things, such as using Yarn, NuGet, or Homebrew. However, consider the possibility that whoever is reading your README is a novice and would like more guidance. Listing specific steps helps remove ambiguity and gets people to using your project as quickly as possible. If it only runs in a specific context like a particular programming language version or operating system or has dependencies that have to be installed manually, also add a Requirements subsection.
Transform date formats (simple date, date with time zone, and timestamp).
```
{
"string_to_date": {
"$.input": "$.string_to_date",
"input_format": "dd-MM-yyyy HH:mm:ss.SSS",
"type": "date",
"format": "dd.MM.yyyy"
},
"zoneddate_to_date": {
"$.input": "$.zoneddate_to_date",
"input_format": "yyyy-MM-dd HH:mm:ss a z",
"type": "date",
"format": "dd.MM.yyyy"
},
"timestamp_to_date": {
"$.input": "$.timestamp_to_date",
"input_format": "timestamp",
"type": "date",
"format": "dd.MM.yyyy HH:mm"
}
}
```
## Usage
Use examples liberally, and show the expected output if you can. It's helpful to have inline the smallest example of usage that you can demonstrate, while providing links to more sophisticated examples if they are too long to reasonably include in the README.
Integer json to string json and vice versa
```
{
"int_to_string": {
"$.input": "$.int_to_string",
"type": "string"
},
"string_to_int": {
"$.input": "$.string_to_int",
"type": "integer"
}
}
```
## Support
Tell people where they can go to for help. It can be any combination of an issue tracker, a chat room, an email address, etc.
String json to decimal json, decimal json with scale,
Decimal json to string json or integer json
## Roadmap
If you have ideas for releases in the future, it is a good idea to list them in the README.
```
{
"str_to_decimal": {
"$.input": "$.str_to_decimal",
"type": "decimal"
},
"str_to_decimal_scale": {
"$.input": "$.str_to_decimal",
"type": "decimal",
"scale": 2
},
"decimal_to_str": {
"$.input": "$.decimal_to_str",
"type": "string"
},
"decimal_to_integer": {
"$.input": "$.decimal_to_integer",
"type": "integer"
}
}
```
String json to boolean and vice versa
```
{
"str_to_bln": {
"$.input": "$.str_to_bln",
"type": "boolean"
},
"bln_to_str": {
"$.input": "$.bln_to_str",
"type": "string"
}
}
```
## Contributing
State if you are open to contributions and what your requirements are for accepting them.
#####/v2/{code} endpoint
For people who want to make changes to your project, it's helpful to have some documentation on how to get started. Perhaps there is a script that they should run or some environment variables that they need to set. Make these steps explicit. These instructions could also be useful to your future self.
Andmemuundur v2 endpoint has the same puropse as v1 but the core logic differs greatly when
writing mapping files.
You can also document commands to lint the code or run tests. These steps help to ensure high code quality and reduce the likelihood that the changes inadvertently break something. Having instructions for running tests is especially helpful if it requires external setup, such as starting a Selenium server for testing in a browser.
The endpoint uses Handlebars logic to write mapping files. Further documentation can be found in
handlebars documentation which at the moment of writing can be found in [here](https://github.com/jknack/handlebars.java).
## Authors and acknowledgment
Show your appreciation to those who have contributed to the project.
Mapping files are located at templates/v2 directory.
## License
For open source projects, say how it is licensed.
## Licence
## Project status
If you have run out of energy or time for your project, put a note at the top of the README saying that development has slowed down or stopped completely. Someone may choose to fork your project or volunteer to step in as a maintainer or owner, allowing your project to keep going. You can also make an explicit request for maintainers.
See licence [here](LICENCE.md).
\ No newline at end of file
### 1. Display output as an array
Array is controlled by the "output_as_array" parameter in the mapping file. In this case "registrikood" is output as array (if more than 1 value exists in response).
* Request:
```
curl -X POST \
http://10.0.24.148:8080/andmemuundur/json/v1/x \
-H 'Content-Type: application/json' \
-H 'cache-control: no-cache' \
-d '{
"data": {
"response": {
"oigused": {
"oigus": [
{
"oigusNimi": "GENERAL.ARENDUS",
"registrikood": "70006317"
},
{
"registrikood": "70006318",
"oigusNimi": "GENERAL.ADMIN"
}
]
}
},
"error": null
}
}'
```
output
```
{
"response": {
"registrikood": ["70006317", "70006318"]
}
}
```
### 2. Filter output values
Only results containing values "LAPS" and "REGISTRIS" will be shown.
* Request:
```
curl -X POST \
http://10.0.24.148:8080/andmemuundur/json/v1/x \
-H 'Content-Type: application/json' \
-H 'cache-control: no-cache' \
-d '{
"data": {
"response": {
"Suhted": {
"Suhteandmed": [
{
"Suhteandmed.ssuhtetyyp": "LAPS",
"Suhteandmed.sIsikukood": "1234",
"Suhteandmed.sStaatus": "REGISTRIS"
},
{
"Suhteandmed.ssuhtetyyp": "VEND",
"Suhteandmed.sIsikukood": "12345",
"Suhteandmed.sStaatus": "REGISTRIS"
},
{
"Suhteandmed.ssuhtetyyp": "LAPS",
"Suhteandmed.sIsikukood": "123456",
"Suhteandmed.sStaatus": "ARHIIVIS"
}
]
}
}
}
}'
```
output
```
{
"response": {
"i18n_minu_lapsed": [{
"i18n_suhtetyyp": "LAPS",
"i18n_isikukood": "1234",
"i18n_staatus": "REGISTRIS"
}]
}
}
```
### 3. JSON transformations
* Mapping: ```template_examples/transformType.json```
To test this example the sample template `transformType.json` from template_examples/ directory should be placed in the templates directory and the service should be restarted.
* Request:
```
curl -X POST \
http://10.0.24.148:8080/andmemuundur/json/v1/transformType \
-H 'Content-Type: application/json' \
-H 'cache-control: no-cache' \
-d '{
"data": {
"string_to_date": "12-02-2018 01:34:00.000",
"zoneddate_to_date": "2018-02-12 10:15:30 AM +05:30",
"timestamp_to_date": "1597659036",
"int_to_string": 123,
"string_to_int": "123",
"str_to_decimal": "123.1234",
"decimal_to_str": 123.1234,
"decimal_to_integer": 123.5234,
"str_to_bln": "true",
"bln_to_str": false
}
}
'
```
output
```
{
"string_to_date": "12.02.2018",
"zoneddate_to_date": "12.02.2018",
"timestamp_to_date": "17.08.2020 13:10",
"int_to_string": "123",
"string_to_int": 123,
"str_to_decimal": 123.1234,
"decimal_to_str": "123.1234",
"decimal_to_integer": 124,
"str_to_bln": true,
"bln_to_str": "false"
}
```
### 4. String replacement
Notice the replaced email addresses and the removed `<img>` tag in the response.
* Request:
```
curl -X POST \
http://10.0.24.148:8080/andmemuundur/json/v1/x \
-H 'Content-Type: application/json' \
-H 'cache-control: no-cache' \
-d '{
"data": {
"summary": "Summary of the message",
"messages": [
{
"from": {
"sent_from_email": "email@inbox.www.eesti.ee",
"sent_from_name": "Tester Dude"
},
"content": {
"subject": "Subject",
"content": {
"ISO-8859-1": {
"text_plain": "This should remain. <img and this img tag should be gone>"
}
}
}
}
]
}
}
'
```
output
```
{
"summary": "Summary of the message",
"messages": [{
"status": {},
"from": {
"i18n_sent_from_email": "email@eesti.ee",
"i18n_sent_from_name": "Tester Dude"
},
"content": {
"i18n_subject": "Subject"
},
"attachments": {}
}]
}
```
### 5. Encoding
Notice the returned value in `i18n_subject`.
* Request:
```
curl -X POST \
http://10.0.24.148:8080/andmemuundur/json/v1/x \
-H 'Content-Type: application/json' \
-H 'cache-control: no-cache' \
-d '{
"data": {
"summary": "Summary of the message",
"messages": [
{
"content": {
"subject": "äöü",
"content": {
"ISO-8859-1": {
"text_plain": "Text"
}
}
}
}
]
}
}
'
```
output
```
{
"summary": "Summary of the message",
"messages": [{
"status": {},
"from": {},
"content": {
"i18n_subject": "äöü"
},
"attachments": {}
}]
}
```
### 6. Sorting
Notice how the messages are returned in a different order based on the `time_received` values.
* Request:
```
curl -X POST \
http://10.0.24.148:8080/andmemuundur/json/v1/x \
-H 'Content-Type: application/json' \
-H 'cache-control: no-cache' \
-d '{
"data": {
"summary": "Summary of the data",
"messages": [
{
"status": {
"time_received": "2019-08-13T17:44Z"
},
"content": {
"subject": "Subject 1",
"content": {
"ISO-8859-1": {
"text_plain": "Text"
}
}
}
}, {
"status": {
"time_received": "2019-08-11T17:44Z"
},
"content": {
"subject": "Subject 2",
"content": {
"ISO-8859-1": {
"text_plain": "Text"
}
}
}
}, {
"status": {
"time_received": "2019-08-12T17:44Z"
},
"content": {
"subject": "Subject 3",
"content": {