Commit 2dc3863f authored by Vitali Stupin's avatar Vitali Stupin
Browse files

Merge pull request #1 in XTIIM/csapi from develop to release

* commit 'deb3decf':
  Update README
  Updated readme
  Add license
  Updated readme
  Adding tests, coverage and lint
  Adding tests
  Adding tests
  Changing deprecated logger.warn
  Refactoring
  Fixing test
  Testing faulty unittest
  First test
  Corrected readme
  Cleanup
parents 6e79f913 deb3decf
Pipeline #14 failed with stages
The MIT License
Copyright (c) 2019 Estonian Information System Authority (RIA),
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
# Central Server API
## Developing
This API is used to add new X-Road members directly to X-Road Central Server without web admin interface.
In order to develop on your machine and test on central server you need to add your public SSH key to the server:
An official management API will be included in a future X-Road release. You can find the roadmap for X-Road [here](https://www.niis.org/xroad-roadmap).
**NB! Make sure your API is not accessible from public internet, and is properly secured in your internal network!**
## API description
API is described using OpenAPI specification: [openapi-definition.yaml](openapi-definition.yaml)
## Installation
Installation was tested with Ubuntu 18.04.
### Program
Provided systemd and nginx configurations assume than program files are installed under `/opt/csapi`. Program is running under `xroad` user to be able to access X-Road configuration files and database without any additional configurations.
Create `/opt/csapi` directory:
```bash
sudo mkdir -p /opt/csapi
```
And copy files `member.py`, `server.py`, and `requirements.txt` into `/opt/csapi` directory.
You will need to install support for python venv:
```bash
ssh-keygen
ssh-copy-id riajenk@jan-center2.ci.kit
sudo apt-get install python3-venv
```
Prepare server:
Then install required python modules into venv:
```bash
sudo apt-get install python3.4-venv
cd /opt/csapi
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
```
Then create a "Run Configuration" that executes `remote_exec.sh`
### Systemd configuration
Testing:
Add service description `systemd/csapi.service` to `/lib/systemd/system/csapi.service`. Then start and enable automatic startup:
```bash
curl -i -d '{"member_code": "XX000002", "member_name": "XX Test 2", "member_class": "GOV"}' -X POST jan-center2.ci.kit:5444/member
sudo systemctl daemon-reload
sudo systemctl start csapi
sudo systemctl enable csapi
```
## API description
API is described using OpenAPI specification: [openapi-definition.yaml](openapi-definition.yaml)
### Nginx configuration
## Nginx configuration
Add nginx configuration from this repository: `nginx/csapi.conf` to nginx server: `/etc/nginx/sites-enabled/csapi.conf`
Create a certificate for nginx (already installed to Central Server):
Create a certificate for nginx (installed by default in X-Road Central Server):
```bash
mkdir -p /etc/nginx/csapi
sudo mkdir -p /etc/nginx/csapi
cd /etc/nginx/csapi
sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout csapi.key -out csapi.crt
```
Cert info:
Cert info (NB! CN should be the domain name of your central server):
```
Country Name (2 letter code) [AU]:EE
State or Province Name (full name) [Some-State]:Harjumaa
Locality Name (eg, city) []:Tallinn
Organization Name (eg, company) [Internet Widgits Pty Ltd]:RIA
Organizational Unit Name (eg, section) []:CSAPI
Common Name (e.g. server FQDN or YOUR name) []:jan-center2.ci.kit
Common Name (e.g. server FQDN or YOUR name) []:central-server.domain.local
Email Address []:
```
Make sure key is accessible to nginx:
```bash
sudo chmod 640 /etc/nginx/csapi/csapi.key
sudo chgrp www-data /etc/nginx/csapi/csapi.key
sudo chmod g+r /etc/nginx/csapi/csapi.key
```
On client side (XTSS app):
......@@ -61,25 +86,58 @@ Country Name (2 letter code) [AU]:EE
State or Province Name (full name) [Some-State]:Harjumaa
Locality Name (eg, city) []:Tallinn
Organization Name (eg, company) [Internet Widgits Pty Ltd]:RIA
Organizational Unit Name (eg, section) []:xtss
Organizational Unit Name (eg, section) []:APIClient
Common Name (e.g. server FQDN or YOUR name) []:
Email Address []:
```
Copy client.srt to nginx machine: `/etc/nginx/csapi/client.crt`
Copy client.crt to Central Server machine: `/etc/nginx/csapi/client.crt`
For testing copy nginx csapi.crt to client and issue curl command:
For testing copy nginx `csapi.crt` to client and issue curl command:
```bash
curl --cert client.crt --key client.key --cacert csapi.crt -i -d '{"member_code": "XX000003", "member_name": "XX Test 3", "member_class": "GOVXXX"}' -X POST https://jan-center2.ci.kit:5443/member
curl --cert client.crt --key client.key --cacert csapi.crt -i -d '{"member_code": "XX000003", "member_name": "XX Test 3", "member_class": "GOVXXX"}' -X POST https://central-server.domain.local:5443/member
```
Add nginx configuration from this repository: `nginx/csapi` to nginx server: `/etc/nginx/sites-enabled/csapi`
Note that you can allow multiple clients (or nodes) by creating certificate bundle. That can be done by concatenating multiple client certificates into single `client.crt` file.
## Testing
## Systemd
Note that `server.py` is a configuration file for logging and Flask and therefore not covered by tests.
Add service description `csapi.service` to `/etc/systemd/system/csapi.service`. Then start and enable automatic startup:
Running the tests:
```bash
sudo systemctl daemon-reload
sudo systemctl start csapi
sudo systemctl enable csapi
cd <project_directory>
python -m unittest
```
Or alternatively run the test file directly:
```bash
python test_member.py
```
In order to measure code coverage install `coverage` module:
```bash
pip install coverage
```
Then run coverage analyse:
```bash
coverage run test_member.py
coverage report member.py
```
Alternatively you can generate html report with:
```bash
coverage run test_member.py
coverage html member.py
```
In order to lint the code install `pylint` module:
```bash
pip install pylint
```
Then run the analyse:
```bash
pylint member.py
```
#!/usr/bin/env python3
"""This is a module for X-Road Central Server member API.
This module allows adding new members to the X-Road Central Server.
"""
import logging
import psycopg2
import re
import psycopg2
from flask import request, jsonify
from flask_restful import Resource
logger = logging.getLogger('member')
DB_CONF_FILE = '/etc/xroad/db.properties'
LOGGER = logging.getLogger('member')
def add_member(member_code, member_name, member_class, json_data):
def get_db_conf():
"""Get Central Server database configuration parameters"""
conf = {
'database': '',
'username': '',
......@@ -18,47 +25,43 @@ def add_member(member_code, member_name, member_class, json_data):
# Getting database credentials from X-Road configuration
try:
with open('/etc/xroad/db.properties', 'r') as dbConf:
for line in dbConf:
m = re.match('^database\\s*=\\s*(.+)$', line)
if m:
conf['database'] = m.group(1)
m = re.match('^username\\s*=\\s*(.+)$', line)
if m:
conf['username'] = m.group(1)
m = re.match('^password\\s*=\\s*(.+)$', line)
if m:
conf['password'] = m.group(1)
with open(DB_CONF_FILE, 'r') as db_conf:
for line in db_conf:
match_res = re.match('^database\\s*=\\s*(.+)$', line)
if match_res:
conf['database'] = match_res.group(1)
match_res = re.match('^username\\s*=\\s*(.+)$', line)
if match_res:
conf['username'] = match_res.group(1)
match_res = re.match('^password\\s*=\\s*(.+)$', line)
if match_res:
conf['password'] = match_res.group(1)
except IOError:
pass
if not conf['username'] or not conf['password'] or not conf['database']:
logger.error('DB_CONF_ERROR: Cannot access database configuration')
return {
'http_status': 500, 'code': 'DB_CONF_ERROR',
'msg': 'Cannot access database configuration'}
return conf
conn = psycopg2.connect(
def get_db_connection(conf):
"""Get connection object for Central Server database"""
return psycopg2.connect(
'host={} port={} dbname={} user={} password={}'.format(
'localhost', '5432', conf['database'], conf['username'], conf['password']))
cur = conn.cursor()
# Check if Member Class is valid
def get_member_class_id(cur, member_class):
"""Get ID of member class from Central Server"""
cur.execute("""select id from member_classes where code=%(str)s""", {'str': member_class})
rec = cur.fetchone()
if rec and len(rec) > 0:
class_id = rec[0]
else:
logger.warn(
'INVALID_MEMBER_CLASS: Provided Member Class does not exist (Request: {})'.format(
json_data))
return {
'http_status': 400, 'code': 'INVALID_MEMBER_CLASS',
'msg': 'Provided Member Class does not exist'}
if rec:
return rec[0]
return None
# Check if Member (Member Code) already exists
def member_exists(cur, member_code):
"""Check if member exists in Central Server"""
cur.execute(
"""
select exists(
......@@ -67,17 +70,21 @@ def add_member(member_code, member_name, member_class, json_data):
)
""", {'str': member_code})
rec = cur.fetchone()
if rec[0] is True:
logger.warn(
'MEMBER_EXISTS: Provided Member already exists (Request: {})'.format(json_data))
return {
'http_status': 409, 'code': 'MEMBER_EXISTS',
'msg': 'Provided Member already exists'}
return rec[0]
# Timestamps must be in UTC timezone
def get_utc_time(cur):
"""Get current time in UTC timezone from Central Server database"""
cur.execute("""select current_timestamp at time zone 'UTC'""")
utc_time = cur.fetchone()[0]
return cur.fetchone()[0]
def add_identifier(cur, **kwargs):
"""Add new X-Road identifier to Central Server
Required keyword arguments:
member_class, member_code, utc_time
"""
cur.execute(
"""
insert into identifiers (
......@@ -87,10 +94,19 @@ def add_member(member_code, member_name, member_class, json_data):
'MEMBER', (select value from system_parameters where key='instanceIdentifier'),
%(class)s, %(code)s, 'ClientId', %(time)s, %(time)s
) returning id
""", {'class': member_class, 'code': member_code, 'time': utc_time}
""", {
'class': kwargs['member_class'], 'code': kwargs['member_code'],
'time': kwargs['utc_time']}
)
identifier_id = cur.fetchone()[0]
return cur.fetchone()[0]
def add_client(cur, **kwargs):
"""Add new X-Road client to Central Server
Required keyword arguments:
member_code, member_name, class_id, identifier_id, utc_time
"""
cur.execute(
"""
insert into security_server_clients (
......@@ -100,11 +116,19 @@ def add_member(member_code, member_name, member_class, json_data):
%(time)s
)
""", {
'code': member_code, 'name': member_name, 'class_id': class_id,
'identifier_id': identifier_id, 'time': utc_time
'code': kwargs['member_code'], 'name': kwargs['member_name'],
'class_id': kwargs['class_id'], 'identifier_id': kwargs['identifier_id'],
'time': kwargs['utc_time']
}
)
def add_client_name(cur, **kwargs):
"""Add new X-Road client name to Central Server
Required keyword arguments:
member_name, identifier_id, utc_time
"""
cur.execute(
"""
insert into security_server_client_names (
......@@ -112,67 +136,116 @@ def add_member(member_code, member_name, member_class, json_data):
) values (
%(name)s, %(identifier_id)s, %(time)s, %(time)s
)
""", {'name': member_name, 'identifier_id': identifier_id, 'time': utc_time}
""", {
'name': kwargs['member_name'], 'identifier_id': kwargs['identifier_id'],
'time': kwargs['utc_time']}
)
cur.close()
conn.commit()
conn.close()
logger.info('Added new member: member_code={}, member_name={}, member_class={}'.format(
member_code, member_name, member_class))
def add_member(member_code, member_name, member_class, json_data):
"""Add new X-Road member to Central Server"""
conf = get_db_conf()
if not conf['username'] or not conf['password'] or not conf['database']:
LOGGER.error('DB_CONF_ERROR: Cannot access database configuration')
return {
'http_status': 500, 'code': 'DB_CONF_ERROR',
'msg': 'Cannot access database configuration'}
with get_db_connection(conf) as conn:
with conn.cursor() as cur:
class_id = get_member_class_id(cur, member_class)
if class_id is None:
LOGGER.warning(
'INVALID_MEMBER_CLASS: Provided Member Class does not exist '
'(Request: %s)', json_data)
return {
'http_status': 400, 'code': 'INVALID_MEMBER_CLASS',
'msg': 'Provided Member Class does not exist'}
if member_exists(cur, member_code):
LOGGER.warning(
'MEMBER_EXISTS: Provided Member already exists '
'(Request: %s)', json_data)
return {
'http_status': 409, 'code': 'MEMBER_EXISTS',
'msg': 'Provided Member already exists'}
# Timestamps must be in UTC timezone
utc_time = get_utc_time(cur)
identifier_id = add_identifier(
cur, member_class=member_class, member_code=member_code, utc_time=utc_time)
add_client(
cur, member_code=member_code, member_name=member_name, class_id=class_id,
identifier_id=identifier_id, utc_time=utc_time)
add_client_name(
cur, member_name=member_name, identifier_id=identifier_id, utc_time=utc_time)
conn.commit()
LOGGER.info(
'Added new member: member_code=%s, member_name=%s, member_class=%s',
member_code, member_name, member_class)
return {'http_status': 201, 'code': 'CREATED', 'msg': 'New member added'}
def make_response(data):
"""Create JSON response object"""
response = jsonify({'code': data['code'], 'msg': data['msg']})
response.status_code = data['http_status']
logger.info('Response: {}'.format(data))
LOGGER.info('Response: %s', data)
return response
def get_input(json_data, param_name):
"""Get parameter from request parameters
Returns two items:
* parameter value
* error response (if parameter not found).
If one parameter is set then other is always None.
"""
try:
param = json_data[param_name]
except KeyError:
LOGGER.warning(
'MISSING_PARAMETER: Request parameter %s is missing '
'(Request: %s)', param_name, json_data)
return None, {
'http_status': 400, 'code': 'MISSING_PARAMETER',
'msg': 'Request parameter {} is missing'.format(param_name)}
return param, None
class MemberApi(Resource):
"""Member API class for Flask"""
@staticmethod
def post():
"""POST method"""
json_data = request.get_json(force=True)
logger.info('Incoming request: {}'.format(json_data))
LOGGER.info('Incoming request: %s', json_data)
try:
member_code = json_data['member_code']
except KeyError:
logger.warn(
'MISSING_PARAMETER: Request parameter member_code is missing '
'(Request: {})'.format(json_data))
return make_response({
'http_status': 400, 'code': 'MISSING_PARAMETER',
'msg': 'Request parameter member_code is missing'})
(member_code, fault_response) = get_input(json_data, 'member_code')
if member_code is None:
return make_response(fault_response)
try:
member_name = json_data['member_name']
except KeyError:
logger.warn(
'MISSING_PARAMETER: Request parameter member_name is missing '
'(Request: {})'.format(json_data))
return make_response({
'http_status': 400, 'code': 'MISSING_PARAMETER',
'msg': 'Request parameter member_name is missing'})
(member_name, fault_response) = get_input(json_data, 'member_name')
if member_name is None:
return make_response(fault_response)
try:
member_class = json_data['member_class']
except KeyError:
logger.warn(
'MISSING_PARAMETER: Request parameter member_class is missing '
'(Request: {})'.format(json_data))
return make_response({
'http_status': 400, 'code': 'MISSING_PARAMETER',
'msg': 'Request parameter member_class is missing'})
(member_class, fault_response) = get_input(json_data, 'member_class')
if member_class is None:
return make_response(fault_response)
try:
response = add_member(member_code, member_name, member_class, json_data)
except psycopg2.Error:
logger.error('DB_ERROR: Unclassified database error')
LOGGER.error('DB_ERROR: Unclassified database error')
response = {
'http_status': 500, 'code': 'DB_ERROR',
'msg': 'Unclassified database error'}
......
#!/usr/bin/env bash
set -e
if [[ ! -d "venv" ]]; then
python3 -m venv venv > /dev/null
fi
source venv/bin/activate
pip install -r requirements.txt > /dev/null
#!/usr/bin/env bash
set -e
SERVER=riajenk@jan-center2.ci.kit
ssh $SERVER "mkdir -p project"
rsync -r -p -l --exclude=\.* --exclude=venv . ${SERVER}:project/
ssh $SERVER "cd project; ./prepare_venv.sh; sudo -u xroad ./run_in_venv.sh"
psycopg2-binary==2.8.2
Flask==1.0.3
Flask-RESTful==0.3.7
gunicorn==19.9.0
coverage
pylint
#!/usr/bin/env bash
set -e
source venv/bin/activate
# Flask (Werkzeug) server
#python server_dev.py
# Production ready Gunicorn server
gunicorn -w 4 -b 127.0.0.1:5444 server:app
#!/usr/bin/env python3
import logging
from flask import Flask, request
from flask_restful import Resource, Api
from member import MemberApi
# NB! For developing only
class StopApi(Resource):
@staticmethod
def get():
func = request.environ.get('werkzeug.server.shutdown')
if func is None:
raise RuntimeError('Not running with the Werkzeug Server')
func()
return 'Server shutting down...'
handler = logging.FileHandler('/var/log/xroad/csapi.log')
handler.setFormatter(logging.Formatter('%(asctime)s - %(process)d - %(levelname)s: %(message)s'))
# Member module logger
logger_m = logging.getLogger('member')
logger_m.setLevel(logging.INFO)
logger_m.addHandler(handler)
# Application logger
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
logger.addHandler(handler)
app = Flask(__name__)
api = Api(app)
api.add_resource(MemberApi, '/member')
logger.info('Starting Central Server API')
if __name__ == '__main__':
# Flask logger
logger_f = logging.getLogger('werkzeug')
logger_f.setLevel(logging.INFO)
logger_f.addHandler(handler)
# Running Flask (Werkzeug) server for development
api.add_resource(StopApi, '/stop')
app.run(debug=False, host='0.0.0.0', port=5444)
This diff is collapsed.
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment