Commit 37d28886 authored by Taavi Meinberg's avatar Taavi Meinberg
Browse files

Pull request #6: Release

Merge in XTIIM/csapi from release to master

* commit '562d3b51':
  Updating requirements
  Updating readme with status API info
  Updating nginx conf
  Adding status API
  Work in progress
  README.md edited online with Bitbucket
  README.md edited online with Bitbucket
  Adding unit tests, update readme
  Adding Client DN check
  Update readme
  Update openapi definition
  Adding tests, refactor, bugfix
  Rename add_identifier->add_member_identifier, add_client->add_member_client
  Adding tests, bugfix
  Subsystem adding
  Rename tests file
  Rename module
parents 0a24a1a9 562d3b51
......@@ -2,3 +2,4 @@
venv*/
.coverage
htmlcov/
config.json
......@@ -18,12 +18,14 @@ Installation was tested with Ubuntu 18.04.
Provided systemd and nginx configurations assume than program files are installed under `/opt/csapi`. Program is running under `xroad` user to be able to access X-Road configuration files and database without any additional configurations.
Create `/opt/csapi` directory:
Create `/opt/csapi` and `/opt/csapi/socket` directories:
```bash
sudo mkdir -p /opt/csapi
sudo mkdir -p /opt/csapi/socket
sudo chown xroad /opt/csapi/socket/
```
And copy files `member.py`, `server.py`, and `requirements.txt` into `/opt/csapi` directory.
And copy files `csapi.py`, `server.py`, and `requirements.txt` into `/opt/csapi` directory.
You will need to install support for python venv:
```bash
......@@ -38,6 +40,8 @@ source venv/bin/activate
pip install -r requirements.txt
```
Create configuration file `/opt/csapi/config.json` based on example configuration `example-config.json`. You need to either set parameter "allow_all" to "true" to disable client certificate check or specify list of trusted Client DN's. Disabled check means that all certificates trusted by Nginx would be allowed.
### Systemd configuration
Add service description `systemd/csapi.service` to `/lib/systemd/system/csapi.service`. Then start and enable automatic startup:
......@@ -93,13 +97,20 @@ Email Address []:
Copy client.crt to Central Server machine: `/etc/nginx/csapi/client.crt`
For testing copy nginx `csapi.crt` to client and issue curl command:
For testing copy nginx `csapi.crt` to client and issue curl commands:
```bash
curl --cert client.crt --key client.key --cacert csapi.crt -i -d '{"member_code": "XX000003", "member_name": "XX Test 3", "member_class": "GOVXXX"}' -X POST https://central-server.domain.local:5443/member
curl --cert client.crt --key client.key --cacert csapi.crt -i -d '{"member_class": "GOVXXX", "member_code": "XX000003", "member_name": "XX Test 3"}' -X POST https://central-server.domain.local:5443/member
curl --cert client.crt --key client.key --cacert csapi.crt -i -d '{"member_class": "GOVXXX", "member_code": "XX000003", "subsystem_code": "SystemXX"}' -X POST https://central-server.domain.local:5443/subsystem
```
Note that you can allow multiple clients (or nodes) by creating certificate bundle. That can be done by concatenating multiple client certificates into single `client.crt` file.
### API Status
API Status is available on `/status` endpoint. You can test that with curl:
```bash
curl -k https://central-server.domain.local:5443/status
```
## Testing
Note that `server.py` is a configuration file for logging and Flask and therefore not covered by tests.
......@@ -112,7 +123,7 @@ python -m unittest
Or alternatively run the test file directly:
```bash
python test_member.py
python test_csapi.py
```
In order to measure code coverage install `coverage` module:
......@@ -122,14 +133,14 @@ pip install coverage
Then run coverage analyse:
```bash
coverage run test_member.py
coverage report member.py
coverage run test_csapi.py
coverage report csapi.py
```
Alternatively you can generate html report with:
```bash
coverage run test_member.py
coverage html member.py
coverage run test_csapi.py
coverage html csapi.py
```
In order to lint the code install `pylint` module:
......@@ -139,5 +150,5 @@ pip install pylint
Then run the analyse:
```bash
pylint member.py
pylint csapi.py
```
#!/usr/bin/env python3
"""This is a module for X-Road Central Server member API.
"""This is a module for X-Road Central Server API.
This module allows adding new members to the X-Road Central Server.
This module allows:
* adding new member to the X-Road Central Server.
* adding new subsystem to the X-Road Central Server.
"""
import json
import logging
import re
import psycopg2
......@@ -12,7 +15,7 @@ from flask import request, jsonify
from flask_restful import Resource
DB_CONF_FILE = '/etc/xroad/db.properties'
LOGGER = logging.getLogger('member')
LOGGER = logging.getLogger('csapi')
def get_db_conf():
......@@ -60,17 +63,32 @@ def get_member_class_id(cur, member_class):
return None
def member_exists(cur, member_code):
"""Check if member exists in Central Server"""
def subsystem_exists(cur, member_id, subsystem_code):
"""Check if subsystem exists in Central Server"""
cur.execute(
"""
select exists(
select * from security_server_clients
where type='XRoadMember' and member_code=%(str)s
where type='Subsystem' and xroad_member_id=%(member_id)s
and subsystem_code=%(subsystem_code)s
)
""", {'str': member_code})
""", {'member_id': member_id, 'subsystem_code': subsystem_code})
return cur.fetchone()[0]
def get_member_data(cur, class_id, member_code):
"""Get member data from Central Server"""
cur.execute(
"""
select id, name
from security_server_clients
where type='XRoadMember' and member_class_id=%(class_id)s
and member_code=%(member_code)s
""", {'class_id': class_id, 'member_code': member_code})
rec = cur.fetchone()
return rec[0]
if rec:
return {'id': rec[0], 'name': rec[1]}
return None
def get_utc_time(cur):
......@@ -79,8 +97,8 @@ def get_utc_time(cur):
return cur.fetchone()[0]
def add_identifier(cur, **kwargs):
"""Add new X-Road identifier to Central Server
def add_member_identifier(cur, **kwargs):
"""Add new X-Road member identifier to Central Server
Required keyword arguments:
member_class, member_code, utc_time
......@@ -101,8 +119,30 @@ def add_identifier(cur, **kwargs):
return cur.fetchone()[0]
def add_client(cur, **kwargs):
"""Add new X-Road client to Central Server
def add_subsystem_identifier(cur, **kwargs):
"""Add new X-Road subsystem identifier to Central Server
Required keyword arguments:
member_class, member_code, subsystem_code, utc_time
"""
cur.execute(
"""
insert into identifiers (
object_type, xroad_instance, member_class, member_code, subsystem_code, type,
created_at, updated_at
) values (
'SUBSYSTEM', (select value from system_parameters where key='instanceIdentifier'),
%(class)s, %(member_code)s, %(subsystem_code)s, 'ClientId', %(time)s, %(time)s
) returning id
""", {
'class': kwargs['member_class'], 'member_code': kwargs['member_code'],
'subsystem_code': kwargs['subsystem_code'], 'time': kwargs['utc_time']}
)
return cur.fetchone()[0]
def add_member_client(cur, **kwargs):
"""Add new X-Road member client to Central Server
Required keyword arguments:
member_code, member_name, class_id, identifier_id, utc_time
......@@ -123,6 +163,27 @@ def add_client(cur, **kwargs):
)
def add_subsystem_client(cur, **kwargs):
"""Add new X-Road subsystem as a client to Central Server
Required keyword arguments:
subsystem_code, member_id, identifier_id, utc_time
"""
cur.execute(
"""
insert into security_server_clients (
subsystem_code, xroad_member_id, server_client_id, type, created_at, updated_at
) values (
%(subsystem_code)s, %(member_id)s, %(identifier_id)s, 'Subsystem', %(time)s,
%(time)s
)
""", {
'subsystem_code': kwargs['subsystem_code'], 'member_id': kwargs['member_id'],
'identifier_id': kwargs['identifier_id'], 'time': kwargs['utc_time']
}
)
def add_client_name(cur, **kwargs):
"""Add new X-Road client name to Central Server
......@@ -142,7 +203,7 @@ def add_client_name(cur, **kwargs):
)
def add_member(member_code, member_name, member_class, json_data):
def add_member(member_class, member_code, member_name, json_data):
"""Add new X-Road member to Central Server"""
conf = get_db_conf()
if not conf['username'] or not conf['password'] or not conf['database']:
......@@ -162,7 +223,7 @@ def add_member(member_code, member_name, member_class, json_data):
'http_status': 400, 'code': 'INVALID_MEMBER_CLASS',
'msg': 'Provided Member Class does not exist'}
if member_exists(cur, member_code):
if get_member_data(cur, class_id, member_code) is not None:
LOGGER.warning(
'MEMBER_EXISTS: Provided Member already exists '
'(Request: %s)', json_data)
......@@ -173,10 +234,10 @@ def add_member(member_code, member_name, member_class, json_data):
# Timestamps must be in UTC timezone
utc_time = get_utc_time(cur)
identifier_id = add_identifier(
identifier_id = add_member_identifier(
cur, member_class=member_class, member_code=member_code, utc_time=utc_time)
add_client(
add_member_client(
cur, member_code=member_code, member_name=member_name, class_id=class_id,
identifier_id=identifier_id, utc_time=utc_time)
......@@ -186,10 +247,71 @@ def add_member(member_code, member_name, member_class, json_data):
conn.commit()
LOGGER.info(
'Added new member: member_code=%s, member_name=%s, member_class=%s',
'Added new Member: member_code=%s, member_name=%s, member_class=%s',
member_code, member_name, member_class)
return {'http_status': 201, 'code': 'CREATED', 'msg': 'New member added'}
return {'http_status': 201, 'code': 'CREATED', 'msg': 'New Member added'}
def add_subsystem(member_class, member_code, subsystem_code, json_data):
"""Add new X-Road subsystem to Central Server"""
conf = get_db_conf()
if not conf['username'] or not conf['password'] or not conf['database']:
LOGGER.error('DB_CONF_ERROR: Cannot access database configuration')
return {
'http_status': 500, 'code': 'DB_CONF_ERROR',
'msg': 'Cannot access database configuration'}
with get_db_connection(conf) as conn:
with conn.cursor() as cur:
class_id = get_member_class_id(cur, member_class)
if class_id is None:
LOGGER.warning(
'INVALID_MEMBER_CLASS: Provided Member Class does not exist '
'(Request: %s)', json_data)
return {
'http_status': 400, 'code': 'INVALID_MEMBER_CLASS',
'msg': 'Provided Member Class does not exist'}
member_data = get_member_data(cur, class_id, member_code)
if member_data is None:
LOGGER.warning(
'INVALID_MEMBER: Provided Member does not exist '
'(Request: %s)', json_data)
return {
'http_status': 400, 'code': 'INVALID_MEMBER',
'msg': 'Provided Member does not exist'}
if subsystem_exists(cur, member_data['id'], subsystem_code):
LOGGER.warning(
'SUBSYSTEM_EXISTS: Provided Subsystem already exists '
'(Request: %s)', json_data)
return {
'http_status': 409, 'code': 'SUBSYSTEM_EXISTS',
'msg': 'Provided Subsystem already exists'}
# Timestamps must be in UTC timezone
utc_time = get_utc_time(cur)
identifier_id = add_subsystem_identifier(
cur, member_class=member_class, member_code=member_code,
subsystem_code=subsystem_code, utc_time=utc_time)
add_subsystem_client(
cur, subsystem_code=subsystem_code, member_id=member_data['id'],
identifier_id=identifier_id, utc_time=utc_time)
add_client_name(
cur, member_name=member_data['name'], identifier_id=identifier_id,
utc_time=utc_time)
conn.commit()
LOGGER.info(
'Added new Subsystem: member_class=%s, member_code=%s, subsystem_code=%s',
member_class, member_code, subsystem_code)
return {'http_status': 201, 'code': 'CREATED', 'msg': 'New Subsystem added'}
def make_response(data):
......@@ -221,14 +343,86 @@ def get_input(json_data, param_name):
return param, None
def load_config(config_file):
"""Load configuration from JSON file"""
try:
with open(config_file, 'r') as conf:
LOGGER.info('Configuration loaded from file "%s"', config_file)
return json.load(conf)
except IOError as err:
LOGGER.error('Cannot load configuration file "%s": %s', config_file, str(err))
return None
except json.JSONDecodeError as err:
LOGGER.error('Invalid JSON configuration file "%s": %s', config_file, str(err))
return None
def check_client(config, client_dn):
"""Check if client dn is in whitelist"""
# If config is None then all clients are not allowed
if config is None:
return False
if config.get('allow_all', False) is True:
return True
allowed = config.get('allowed')
if client_dn is None or not isinstance(allowed, list):
return False
if client_dn in allowed:
return True
return False
def incorrect_client(client_dn):
"""Return error response when client is not allowed"""
LOGGER.error('FORBIDDEN: Client certificate is not allowed: %s', client_dn)
return make_response({
'http_status': 403, 'code': 'FORBIDDEN',
'msg': 'Client certificate is not allowed: {}'.format(client_dn)})
def test_db():
"""Add new X-Road subsystem to Central Server"""
conf = get_db_conf()
if not conf['username'] or not conf['password'] or not conf['database']:
LOGGER.error('DB_CONF_ERROR: Cannot access database configuration')
return {
'http_status': 500, 'code': 'DB_CONF_ERROR',
'msg': 'Cannot access database configuration'}
with get_db_connection(conf) as conn:
with conn.cursor() as cur:
cur.execute("""select 1 from system_parameters where key='instanceIdentifier'""")
rec = cur.fetchone()
if rec:
return {
'http_status': 200, 'code': 'OK',
'msg': 'API is ready'}
return {'http_status': 500, 'code': 'DB_ERROR', 'msg': 'Unexpected DB state'}
class MemberApi(Resource):
"""Member API class for Flask"""
@staticmethod
def post():
def __init__(self, config):
self.config = config
def post(self):
"""POST method"""
json_data = request.get_json(force=True)
client_dn = request.headers.get('X-Ssl-Client-S-Dn')
LOGGER.info('Incoming request: %s', json_data)
LOGGER.info('Client DN: %s', client_dn)
if not check_client(self.config, client_dn):
return incorrect_client(client_dn)
(member_class, fault_response) = get_input(json_data, 'member_class')
if member_class is None:
return make_response(fault_response)
(member_code, fault_response) = get_input(json_data, 'member_code')
if member_code is None:
......@@ -238,14 +432,70 @@ class MemberApi(Resource):
if member_name is None:
return make_response(fault_response)
try:
response = add_member(member_class, member_code, member_name, json_data)
except psycopg2.Error as err:
LOGGER.error('DB_ERROR: Unclassified database error: %s', err)
response = {
'http_status': 500, 'code': 'DB_ERROR',
'msg': 'Unclassified database error'}
return make_response(response)
class SubsystemApi(Resource):
"""Subsystem API class for Flask"""
def __init__(self, config):
self.config = config
def post(self):
"""POST method"""
json_data = request.get_json(force=True)
client_dn = request.headers.get('X-Ssl-Client-S-Dn')
LOGGER.info('Incoming request: %s', json_data)
LOGGER.info('Client DN: %s', client_dn)
if not check_client(self.config, client_dn):
return incorrect_client(client_dn)
(member_class, fault_response) = get_input(json_data, 'member_class')
if member_class is None:
return make_response(fault_response)
(member_code, fault_response) = get_input(json_data, 'member_code')
if member_code is None:
return make_response(fault_response)
(subsystem_code, fault_response) = get_input(json_data, 'subsystem_code')
if subsystem_code is None:
return make_response(fault_response)
try:
response = add_subsystem(member_class, member_code, subsystem_code, json_data)
except psycopg2.Error as err:
LOGGER.error('DB_ERROR: Unclassified database error: %s', err)
response = {
'http_status': 500, 'code': 'DB_ERROR',
'msg': 'Unclassified database error'}
return make_response(response)
class StatusApi(Resource):
"""Status API class for Flask"""
def __init__(self, config):
self.config = config
@staticmethod
def get():
"""GET method"""
LOGGER.info('Incoming status request')
try:
response = add_member(member_code, member_name, member_class, json_data)
except psycopg2.Error:
LOGGER.error('DB_ERROR: Unclassified database error')
response = test_db()
except psycopg2.Error as err:
LOGGER.error('DB_ERROR: Unclassified database error: %s', err)
response = {
'http_status': 500, 'code': 'DB_ERROR',
'msg': 'Unclassified database error'}
......
{
"allow_all": false,
"allowed": [
"OU=xtss,O=RIA,C=EE"
]
}
......@@ -14,11 +14,17 @@ server {
# who fail authentication
ssl_verify_client optional;
location /status {
# No auth required for status
proxy_pass http://unix:/opt/csapi/socket/csapi.sock;
}
location / {
# Require authentication!!!
if ($ssl_client_verify != SUCCESS) {
return 403;
}
proxy_set_header X-SSL-Client-S-DN $ssl_client_s_dn;
proxy_pass http://unix:/opt/csapi/socket/csapi.sock;
}
}
......@@ -25,34 +25,44 @@ paths:
content:
application/json:
schema:
$ref: '#/components/schemas/Response201'
$ref: '#/components/schemas/ResponseMember201'
examples:
created:
summary: Member created
value: {"code": "CREATED", "msg": "New member added"}
value: {"code": "CREATED", "msg": "New Member added"}
'400':
description: Invalid input
content:
application/json:
schema:
$ref: '#/components/schemas/Response400'
$ref: '#/components/schemas/ResponseMember400'
examples:
missingParam:
summary: Required parameter is mussing
summary: Required parameter is missing
value: {"code": "MISSING_PARAMETER", "msg": "Request parameter member_name is missing"}
invalidClass:
summary: Member class is not found in Central Server
value: {"code": "INVALID_MEMBER_CLASS", "msg": "Provided Member Class does not exist"}
'409':
description: An existing member already exists
description: Provided Member already exists
content:
application/json:
schema:
$ref: '#/components/schemas/Response409'
$ref: '#/components/schemas/ResponseMember409'
examples:
memberExists:
summary: Provided Member already exists in Central Server
value: {"code": "MEMBER_EXISTS", "msg": "Provided Member already exists"}
'403':
description: Client certificate is not allowed
content:
application/json:
schema:
$ref: '#/components/schemas/ResponseMember403'
examples:
memberExists:
summary: Client certificate is not allowed
value: {"code": "FORBIDDEN", "msg": "Client certificate is not allowed"}
'500':
description: Server side error
content:
......@@ -73,28 +83,132 @@ paths:
$ref: '#/components/schemas/Member'
examples:
member:
summary: A bar example
value: {"member_code": "00000000", "member_name": "Member 0", "member_class": "GOV"}
summary: Example request parameters
value: {"member_class": "GOV", "member_code": "00000000", "member_name": "Member 0"}
description: New Member to add
/subsystem:
post:
tags:
- admin
summary: add new X-Road Subsystem
operationId: addSubsystem
description: Adds new X-Road Subsystem to Central Server
responses:
'201':
description: Subsystem added
content:
application/json:
schema:
$ref: '#/components/schemas/ResponseSubsystem201'
examples:
created:
summary: Subsystem created
value: {"code": "CREATED", "msg": "New Subsystem added"}
'400':
description: Invalid input
content:
application/json:
schema:
$ref: '#/components/schemas/ResponseSubsystem400'
examples:
missingParam:
summary: Required parameter is mussing
value: {"code": "MISSING_PARAMETER", "msg": "Request parameter member_name is missing"}
invalidClass:
summary: Member class is not found in Central Server
value: {"code": "INVALID_MEMBER_CLASS", "msg": "Provided Member Class does not exist"}
ivalidMember:
summary: Member class is not found in Central Server
value: {"code": "INVALID_MEMBER", "msg": "Provided Member does not exist"}
'403':
description: Client certificate is not allowed
content:
application/json:
schema:
$ref: '#/components/schemas/ResponseSubsystem403'
examples:
memberExists:
summary: Client certificate is not allowed
value: {"code": "FORBIDDEN", "msg": "Client certificate is not allowed"}
'409':
description: Provided Subsystem already exists
content:
application/json:
schema:
$ref: '#/components/schemas/ResponseSubsystem409'
examples: