Commit 3a05d977 authored by Märten Heinsalu's avatar Märten Heinsalu

Initial commit

parents
Pipeline #22 failed with stages
.idea/
*.iml
target/
\ No newline at end of file
# PUMP
## Dependencies
- `Java 8`
- `Oracle DB 12`
- `X-road` (https://ria.ee/et/riigi-infosusteem/andmevahetuskiht-x-tee.html)
- `X-gate` (custom application of Statistics Estonia)
## Configuration
- Specify configuration location in program arguments: `--spring.config.location=config/dev/`
- In `config/dev`, there is an example how to configure application
## DB
### Create
- If you are using multitenant architecture, then username must begin with "c##" in Oracle DB
- Create pump database
```sql
-- If you want to start from scratch:
-- DROP USER c##pump CASCADE;
CREATE USER c##pump IDENTIFIED BY pump_password;
GRANT ALL PRIVILEGES TO c##pump;
```
- You also need x-gate database setup.
```sql
-- If you want to start from scratch:
-- DROP USER c##xgate CASCADE;
CREATE USER c##xgate IDENTIFIED BY xgate_password;
GRANT ALL PRIVILEGES TO c##xgate;
```
- Run `xgate-db-setup.sql` as user `c##xgate`.
### Migrations
- For migrations [Liquibase](https://www.liquibase.org/) is used
- By default application runs migrations.
- If you want to disable migrations then you must set `spring.liquibase.enabled` to `false`.
- Then you can run migrations manually:
```sh
java -jar liquibase.jar \
--driver=oracle.jdbc.OracleDriver \
--classpath=server.war \
--changeLogFile=db/changelog/db.changelog-master.xml \
--url=jdbc:oracle:thin:@localhost:1521:ORCLCDB \
--username=c##pump \
--password=pump_password \
update
```
## Monitoring
Endpoints are visible in `/info` page.
This can be changed in configuration by setting `management.endpoints.web.base-path`.
## Development
- Create database as described before
- Create new application in IntelliJ IDEA:
- Main class: `ee.stat.pump.server.Application`
- Program arguments:
```
--spring.config.location=config/dev/
```
- VM arguments (truststore contains x-road security server certificate of development):
```
-Djavax.net.ssl.trustStore=config/dev/truststore.jks
-Djavax.net.ssl.trustStorePassword=changeit
```
- Include dependencies with "Provided" scope: `checked`
## FYI
- You can create CRON expressions in [cronmaker](http://www.cronmaker.com/)
- You can create truststore like this:
```
keytool -import -keystore truststore.jks -storepass changeit -alias xtee6.dev -file xtee6dev.crt
```
- [Camunda modeler](https://camunda.com/download/modeler/) is used for diagram.
## Template parsing
- [Mustache](https://github.com/spullara/mustache.java) template engine is used
- Variables have 2 main groups `system` and `body`
- System variables contain:
- `uuid` - universally unique identifier. It can be accessed in template by using variable `system.uuid`
- `userId` - from application configuration (`application.message.userId`). It can be accessed in template by using variable `system.userId`
- `xRoadInstance` - from application configuration (`application.message.xRoadInstance`)
- `memberClass` - from application configuration (`application.message.memberClass`)
- `memberCode` - from application configuration (`application.message.memberCode`)
- `subsystemCode` - from application configuration (`application.message.subsystemCode`)
- Local timestamps which formats can be specified in application configuration (`application.message.timestampFormats`)
- These are not mandatory
- You can add as many formats as you want to - they are picked up automatically
- When there is one with wrong format, then it is ignored
- When in configuration file there is `application.message.timestampFormats.datetime` with value `dd.MM.yyyy HH:mm:ss` then
- It can be accessed in template by using variable `system.timestamp_datetime`
- The result will be like `06.09.2018 16:30:28`
- You can use whatever name you like instead of `datetime`
- Body variables contain everything DB-function returns
- If DB-function returns JSON like `{"name": "Fred"}` then in template it can be accessed by using variable `body.name`
# Docker setup
- In `environment` folder there is docker setup for testing.
- Follow instructions to build [Oracle Database 11g Release 2 (11.2.0.2) Express Edition](https://github.com/oracle/docker-images/tree/master/OracleDatabase/SingleInstance)
- Verify that `application.yml` and `truststore.jks` are up to date in `config/docker`
- Run `docker-compose up -d`
- Wait until oracle db is up and running (`docker logs -f oracle`)
- Create databases as described previously
- To connect to database in docker use:
- Hostname: `localhost`
- Port: `1521`
- SID: `XE`
- Username: `SYSTEM`
- Password: `password`
- Copy `server/target/pump-server.war` into `environment/webapps` folder (will be created by docker)
- When you want to stop tomcat, then: `docker-compose stop tomcat`
- When you want to start tomcat, then: `docker-compose start tomcat`
- When you want to restart tomcat, then: `docker-compose restart tomcat`
\ No newline at end of file
spring:
liquibase:
enabled: true
change-log: classpath:/db/changelog/db.changelog-master.xml
url: jdbc:oracle:thin:@localhost:1521:XE
user: c##pump
password: pump_password
datasource:
url: jdbc:oracle:thin:@localhost:1521:XE
username: c##pump
password: pump_password
x-gate-datasource:
url: jdbc:oracle:thin:@localhost:1521:XE
username: c##xgate
password: xgate_password
jpa:
properties:
hibernate.show_sql: false
hibernate.use_sql_comments: true
hibernate.format_sql: true
hibernate.type: trace
logging.file: pump.log
logging:
file:
max-size: 100MB
management:
info:
git:
mode: full
endpoint:
health:
show-details: always
endpoints:
web:
base-path: /info
exposure:
include: health,info
jmx:
exposure:
include: health,info
application:
id: PUMP_1
securityServerUrl: https://xtee6.dev
xGate:
url: https://xtee6.dev
objectType: SERVICE
xRoadInstance: ee-dev
memberClass: GOV
memberCode: 70000332
subsystemCode: estat
serviceCode: SubmitData
serviceVersion: v1
soap:
connectionTimeoutInMilliseconds: 10000
defaultReadTimeoutInMilliseconds: 60000
serviceConfiguration:
updateIntervalInMilliseconds: 5000
cleanupIntervalInMilliseconds: 10000
corePoolSize: 10
maxPoolSize: 15
queueCapacity: 2147483647
waitForTasksToCompleteOnShutdown: false
awaitTerminationSeconds: 0
keepAliveSeconds: 600
ticketBasedRequest:
updateIntervalInMilliseconds: 1000
cleanupIntervalInMilliseconds: 10000
corePoolSize: 10
maxPoolSize: 15
queueCapacity: 2147483647
waitForTasksToCompleteOnShutdown: false
awaitTerminationSeconds: 0
keepAliveSeconds: 600
message:
userId: EE11111111111
objectType: SUBSYSTEM
xRoadInstance: ee-dev
memberClass: COM
memberCode: 10364097
subsystemCode: VTA
timestampFormats:
year: yyyy
date: dd.MM.yyyy
datetime: dd.MM.yyyy HH:mm:ss
namespace:
soapenv: http://schemas.xmlsoap.org/soap/envelope/
iden: http://x-road.eu/xsd/identifiers
xro: http://x-road.eu/xsd/xroad.xsd
xsd: http://www.w3.org/2001/XMLSchema
xsi: http://www.w3.org/2001/XMLSchema-instance
emta: http://emta-v6.x-road.eu
kpr: http://producers.kpr.x-road.eu/producer/kpr
arireg: http://arireg.x-road.eu/producer/
\ No newline at end of file
spring:
liquibase:
enabled: true
change-log: classpath:/db/changelog/db.changelog-master.xml
url: jdbc:oracle:thin:@oracle:1521:XE
user: c##pump
password: pump_password
datasource:
url: jdbc:oracle:thin:@oracle:1521:XE
username: c##pump
password: pump_password
x-gate-datasource:
url: jdbc:oracle:thin:@oracle:1521:XE
username: c##xgate
password: xgate_password
jpa:
properties:
hibernate.show_sql: false
hibernate.use_sql_comments: true
hibernate.format_sql: true
hibernate.type: trace
logging.file: /usr/local/tomcat/logs/pump/pump.log
logging:
file:
max-size: 100KB
management:
info:
git:
mode: full
endpoint:
health:
show-details: always
endpoints:
web:
base-path: /info
exposure:
include: health,info
jmx:
exposure:
include: health,info
application:
id: PUMP_1
securityServerUrl: https://xtee6.dev
xGate:
url: https://xtee6.dev
objectType: SERVICE
xRoadInstance: ee-dev
memberClass: GOV
memberCode: 70000332
subsystemCode: estat
serviceCode: SubmitData
serviceVersion: v1
soap:
connectionTimeoutInMilliseconds: 10000
defaultReadTimeoutInMilliseconds: 60000
serviceConfiguration:
updateIntervalInMilliseconds: 5000
cleanupIntervalInMilliseconds: 10000
corePoolSize: 10
maxPoolSize: 15
queueCapacity: 2147483647
waitForTasksToCompleteOnShutdown: false
awaitTerminationSeconds: 0
keepAliveSeconds: 600
ticketBasedRequest:
updateIntervalInMilliseconds: 1000
cleanupIntervalInMilliseconds: 10000
corePoolSize: 10
maxPoolSize: 15
queueCapacity: 2147483647
waitForTasksToCompleteOnShutdown: false
awaitTerminationSeconds: 0
keepAliveSeconds: 600
message:
userId: EE11111111111
objectType: SUBSYSTEM
xRoadInstance: ee-dev
memberClass: COM
memberCode: 10364097
subsystemCode: VTA
timestampFormats:
year: yyyy
date: dd.MM.yyyy
datetime: dd.MM.yyyy HH:mm:ss
namespace:
soapenv: http://schemas.xmlsoap.org/soap/envelope/
iden: http://x-road.eu/xsd/identifiers
xro: http://x-road.eu/xsd/xroad.xsd
xsd: http://www.w3.org/2001/XMLSchema
xsi: http://www.w3.org/2001/XMLSchema-instance
emta: http://emta-v6.x-road.eu
kpr: http://producers.kpr.x-road.eu/producer/kpr
arireg: http://arireg.x-road.eu/producer/
\ No newline at end of file
This diff is collapsed.
version: '2'
services:
oracle:
container_name: oracle
image: oracle/database:11.2.0.2-xe
shm_size: '1gb'
volumes:
- oracle11-oradata:/u01/app/oracle/oradata
environment:
ORACLE_PWD: password
ports:
- 1521:1521
tomcat:
container_name: tomcat
image: tomcat:8.5-jre8
volumes:
- ./webapps:/usr/local/tomcat/webapps
- ../config/docker/truststore.jks:/opt/pump/config/truststore.jks
- ../config/docker/application.yml:/opt/pump/config/application.yml
- ./pump-server.xml:/usr/local/tomcat/conf/Catalina/localhost/pump-server.xml
environment:
CATALINA_OPTS: >
-Djavax.net.ssl.trustStore=/opt/pump/config/truststore.jks
-Djavax.net.ssl.trustStorePassword=changeit
TZ: Europe/Tallinn
ports:
- 8080:8080
depends_on:
- oracle
volumes:
oracle11-oradata:
\ No newline at end of file
<?xml version="1.0" encoding="UTF-8"?>
<Context>
<Environment name="spring.config.location" value="file:/opt/pump/config/application.yml" type="java.lang.String"/>
</Context>
\ No newline at end of file
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>ee.stat.pump</groupId>
<artifactId>pump</artifactId>
<packaging>pom</packaging>
<version>1.0</version>
<modules>
<module>server</module>
</modules>
<description>Projekt on rahastatud Euroopa Liidu struktuurfondidest.</description>
</project>
\ No newline at end of file
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.1.1.RELEASE</version>
<relativePath/>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>server</artifactId>
<packaging>war</packaging>
<version>1.0</version>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<project.name>pump-server</project.name>
<java.version>1.8</java.version>
<ojdbc6.version>11.2.0.2</ojdbc6.version>
<lombok.version>1.18.4</lombok.version>
<mustache.version>0.9.5</mustache.version>
<git-commit-id-plugin.version>2.2.1</git-commit-id-plugin.version>
<maven-jaxb2-plugin.version>0.14.0</maven-jaxb2-plugin.version>
<axiom.version>1.2.22</axiom.version>
</properties>
<repositories>
<repository>
<id>http://repo.rmv</id>
<name>rmit-maven-repo</name>
<url>http://repo.rmv/nexus/repository/maven-releases</url>
<layout>default</layout>
<releases>
<enabled>true</enabled>
<updatePolicy>always</updatePolicy>
</releases>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>com.github.spullara.mustache.java</groupId>
<artifactId>compiler</artifactId>
<version>${mustache.version}</version>
</dependency>
<dependency>
<groupId>com.oracle</groupId>
<artifactId>ojdbc6</artifactId>
<version>${ojdbc6.version}</version>
</dependency>
<dependency>
<groupId>org.apache.ws.commons.axiom</groupId>
<artifactId>axiom</artifactId>
<version>${axiom.version}</version>
<type>pom</type>
</dependency>
<dependency>
<groupId>org.apache.ws.commons.axiom</groupId>
<artifactId>axiom-impl</artifactId>
<version>${axiom.version}</version>
</dependency>
<dependency>
<groupId>org.liquibase</groupId>
<artifactId>liquibase-core</artifactId>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>${lombok.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jdbc</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-quartz</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-tomcat</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web-services</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.h2database</groupId>
<artifactId>h2</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<finalName>${project.name}</finalName>
</configuration>
</plugin>
<plugin>
<groupId>pl.project13.maven</groupId>
<artifactId>git-commit-id-plugin</artifactId>
<version>${git-commit-id-plugin.version}</version>
<configuration>
<failOnNoGitDirectory>false</failOnNoGitDirectory>
<failOnUnableToExtractRepoInfo>false</failOnUnableToExtractRepoInfo>
<includeOnlyProperties>
<includeOnlyProperty>git.build.time$</includeOnlyProperty>
<includeOnlyProperty>git.commit.id$</includeOnlyProperty>
<includeOnlyProperty>git.build.version$</includeOnlyProperty>
</includeOnlyProperties>
</configuration>
</plugin>
<plugin>
<groupId>org.jvnet.jaxb2.maven2</groupId>
<artifactId>maven-jaxb2-plugin</artifactId>
<version>${maven-jaxb2-plugin.version}</version>
<executions>
<execution>
<goals>
<goal>generate</goal>
</goals>
</execution>
</executions>
<configuration>
<schemaDirectory>src/main/resources/wsdl</schemaDirectory>
<schemaIncludes>
<include>*.wsdl</include>
</schemaIncludes>
<schemaLanguage>WSDL</schemaLanguage>
<generatePackage>xroad.xgate</generatePackage>
</configuration>
</plugin>
</plugins>
</build>
</project>
\ No newline at end of file
package ee.stat.pump.server;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.builder.SpringApplicationBuilder;
import org.springframework.boot.web.servlet.support.SpringBootServletInitializer;
@SpringBootApplication
public class Application extends SpringBootServletInitializer {
public static void main(String[] args) {
SpringApplication.run(Application.class, args);
}
@Override
protected SpringApplicationBuilder configure(SpringApplicationBuilder builder) {
return builder.sources(Application.class);
}
}
package ee.stat.pump.server.config;
import ee.stat.pump.server.service.ServiceConfigurationCleanupService;
import ee.stat.pump.server.service.TicketBasedRequestCleanupService;
import ee.stat.pump.server.service.scheduler.ServiceSchedulerUpdateService;
import ee.stat.pump.server.service.scheduler.TicketBasedRequestSchedulerUpdateService;
import ee.stat.pump.server.service.storage.XGateClient;
import ee.stat.pump.server.xml.XGateMarshaller;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.oxm.jaxb.Jaxb2Marshaller;
import org.springframework.scheduling.annotation.EnableScheduling;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor;
import org.springframework.ws.soap.axiom.AxiomSoapMessageFactory;
import java.util.concurrent.Executor;
@Configuration
@EnableConfigurationProperties({ApplicationProperties.class})
@EnableScheduling
public class ApplicationConfiguration {
private ApplicationProperties applicationProperties;
private ServiceSchedulerUpdateService serviceSchedulerUpdateService;
private ServiceConfigurationCleanupService serviceConfigurationCleanupService;
private TicketBasedRequestSchedulerUpdateService ticketBasedRequestSchedulerUpdateService;
private TicketBasedRequestCleanupService ticketBasedRequestCleanupService;
@Autowired
public ApplicationConfiguration(ApplicationProperties applicationProperties,
ServiceSchedulerUpdateService serviceSchedulerUpdateService,
ServiceConfigurationCleanupService serviceConfigurationCleanupService,
TicketBasedRequestSchedulerUpdateService ticketBasedRequestSchedulerUpdateService,
TicketBasedRequestCleanupService ticketBasedRequestCleanupService) {
this.applicationProperties = applicationProperties;
this.serviceSchedulerUpdateService = serviceSchedulerUpdateService;
this.serviceConfigurationCleanupService = serviceConfigurationCleanupService;
this.ticketBasedRequestSchedulerUpdateService = ticketBasedRequestSchedulerUpdateService;
this.ticketBasedRequestCleanupService = ticketBasedRequestCleanupService;
}
@Bean(name = "serviceTaskExecutor")
public Executor serviceTaskExecutor() {
return getThreadPoolTaskExecutor(applicationProperties.getServiceConfiguration());
}
@Bean(name = "ticketBasedRequestTaskExecutor")
public Executor ticketBasedRequestTaskExecutor() {
return getThreadPoolTaskExecutor(applicationProperties.getTicketBasedRequest());
}
ThreadPoolTaskExecutor getThreadPoolTaskExecutor(ApplicationProperties.ExecutionProperties executionProperties) {
ThreadPoolTaskExecutor threadPoolTaskExecutor = new ThreadPoolTaskExecutor();
threadPoolTaskExecutor.setCorePoolSize(executionProperties.getCorePoolSize());
threadPoolTaskExecutor.setMaxPoolSize(executionProperties.getMaxPoolSize());
threadPoolTaskExecutor.setQueueCapacity(executionProperties.getQueueCapacity());
threadPoolTaskExecutor.setWaitForTasksToCompleteOnShutdown(executionProperties.isWaitForTasksToCompleteOnShutdown());
threadPoolTaskExecutor.setAwaitTerminationSeconds(executionProperties.getAwaitTerminationSeconds());
threadPoolTaskExecutor.setKeepAliveSeconds(executionProperties.getKeepAliveSeconds());
threadPoolTaskExecutor.initialize();
return threadPoolTaskExecutor;
}
@Bean
public Jaxb2Marshaller marshaller() {
Jaxb2Marshaller marshaller = new XGateMarshaller();
marshaller.setContextPath("xroad.xgate");
return marshaller;
}
@Bean
public AxiomSoapMessageFactory messageFactory() throws Exception {
AxiomSoapMessageFactory messageFactory = new AxiomSoapMessageFactory();
messageFactory.afterPropertiesSet();
return messageFactory;
}
@Bean
public XGateClient xGateClient(Jaxb2Marshaller marshaller, AxiomSoapMessageFactory messageFactory) {
XGateClient client = new XGateClient();
client.setDefaultUri(applicationProperties.getXGate().getUrl());
client.setMarshaller(marshaller);
client.setUnmarshaller(marshaller);
client.setMessageFactory(messageFactory);
return client;
}
@Scheduled(initialDelay = 0, fixedDelayString = "${application.serviceConfiguration.updateIntervalInMilliseconds}")
public void updateServiceScheduler() {
serviceSchedulerUpdateService.updateScheduler();
}
@Scheduled(initialDelay = 0, fixedDelayString = "${application.ticketBasedRequest.updateIntervalInMilliseconds}")
public void updateTicketBasedRequestScheduler() {
ticketBasedRequestSchedulerUpdateService.updateScheduler();
}
@Scheduled(initialDelay = 0, fixedDelayString = "${application.serviceConfiguration.cleanupIntervalInMilliseconds}")
public void unlockHaltedServiceConfigurations() {
serviceConfigurationCleanupService.unlockHaltedServiceConfigurations();
}
@Scheduled(initialDelay = 0, fixedDelayString = "${application.ticketBasedRequest.cleanupIntervalInMilliseconds}")