Acra Engineering Examples illustrate the integration of Acra data protection suite into your existing application. Protecting the data is completely transparent for the users and requires minimal changes in the infrastructure.

This collection has several example application. Each folder contains docker-compose file, that describes key management procedures and configurations of Acra.

Integrating Acra into any application requires 3 steps:

1. Generate cryptographic keys. In this examples, we generate only required keys for each example (Master key, and data encryption keys, rarely others). Refer to Key management to learn more about keys.
2. Configure and deploy services.
   1. transparent encryption for SQL databases – configure and deploy AcraServer. Configure AcraServer's behavior, set up TLS, connect to the database, select which fields/columns to encrypt.
   2. encryption-as-a-service for NoSQL databases – configure and deploy AcraTranslator. Configure AcraTranslator's behavior, set up TLS, select gRPC or REST API.
   3. client-side encryption – you can encrypt data in the client application using AcraWriter, then decrypt data on AcraServer or AcraTranslator.
3. Update client-side code.
   1. transparent encryption for SQL databases – just point client-side app to AcraServer instead of the database.
   2. encryption-as-a-service for NoSQL databases – call AcraTranslator API from client-side app and encrypt/decrypt fields on AcraTranslator.
   3. client-side encryption – integrate AcraWriter, call it to encrypt fields in the app before sending them to the database.

Please refer to the Acra Data flows for more detailed description and schemes.

Django web application, transparent encryption/decryption, AcraServer, PostgreSQL.

Follow Integrating AcraServer into infrastructure guide or a tutorial on dev.to How to encrypt database fields transparently for your app using Acra and DigitalOcean managed PostgreSQL.

Transparent encryption mode (server-side encryption and decryption): data is encrypted and decrypted on the AcraServer:

curl https://raw.githubusercontent.com/cossacklabs/acra-engineering-demo/master/run.sh | \
    bash -s -- django-transparent

This command downloads the code of Django website example, Acra Docker containers, PostgreSQL database, Prometheus, Grafana, pgAdmin images and sets up the environment, configures AcraServer to encrypt data, and provides a list of links for you to try.

The client application is the famous Django app example – the source code of djangoproject.com. We've updated their source code to protect blog posts. Application stores blog posts in PosgtreSQL database. We encrypt blog posts' content before storing in database, and decrypt when reading from database.

Django app does not encrypt the sensitive fields, it just passes data to AcraServer through secured TLS channel (which pretends to be a database). AcraServer encrypts these sensitive fields and stores them into database.

Django app reads the decrypted posts from the database through AcraServer.

From the users' perspective, the website works as it used to. However, the blog posts are protected now.

Please add a temporary entry to the hosts file:

echo "$SERVER_IP www.djangoproject.example" >> /etc/hosts

where SERVER_IP is the IP address of the server that is running the Acra Engineering Demo (if you run the demo on your machine, set it to 127.0.0.1). Updating the hosts file is required because we will run the protected djangoproject site locally. You can remove this line when you stop needed to access the demo site.

1. Log into admin cabinet http://www.djangoproject.example:8000/admin/blog/entry/ using user/password: admin/admin.

Add a blog post to the Blogs/Entries:

You can go to Dashboard window http://www.djangoproject.example:8000/admin/dashboard/category/

And add a new Dashboard category:

2. Open the blog posts' feed http://www.djangoproject.example:8000/weblog/ and see your fresh post.

   Also, you can open the dashboard category' feed http://www.djangoproject.example:8000/admin/dashboard/category/ and see your newly created category:

Everything worked well! Now, let's check the content of the database.

Log into the web PostgreSQL interface http://www.djangoproject.example:8008 using user/password: test@test.test/test.

Find your blog post in Servers > postgresql > databases > djangoproject > Schemas > public > Tables > blog_entries and open context menu with right-click.

Dashboard categories are in Servers > postgresql > databases > djangoproject > Schemas > public > Tables > dashboard_category.

Select View/Edit Data > All rows and now you can see content of the table. Download and read the content – it's encrypted.

So, the blog posts/dashboard categories are stored encrypted, but it's transparent for site visitors and admins.

Open Grafana dashboards to see the performance stats of AcraServer. We collect following metrics: the number of decrypted cryptographic containers (AcraStructs and AcraBlocks), request and response processing time.

Grafana is available at http://www.djangoproject.example:3000.

AcraServer can export detailed traces to Jaeger. Use this data to optimize the performance of the entire system.

Jaeger is available at http://www.djangoproject.example:16686.

There's more to explore:

1. PostgreSQL – connect directly to the database using the admin account postgres/test: postgresql://localhost:5432.

2. pgAdmin - connect directly to the database using WebUI and user account login:test@test.test/password:test: http://localhost:8008

3. Prometheus – examine the collected metrics: http://localhost:9090.

4. Grafana – see the dashboards with Acra metrics: http://localhost:3000.

5. Jaeger – view traces: http://localhost:16686.

6. Docker-compose.django.yml file – read details about configuration and containers used in this example.

So, was it easy to integrate Acra into Django application? Sure it was!

1. AcraServer returns binary data, so we wrote simple wrapper classes to perform encoding and decoding data.

2. We changed original fields types to new ones.

3. Created database migration file to convert encrypted fields to binary.

Those are all the code changes! 🎉

Learn how to prevent possible intrusions by adding "poison records" special crafted data to your database and filtering potential attackers who try to access them.

What's inside: Go client application, AcraServer in transparent encryption mode (server-side encryption), PostgreSQL database.

Follow the guide: Acra intrusion detection example.

TimescaleDB, AcraServer in Transparent encryption mode.

curl https://raw.githubusercontent.com/cossacklabs/acra-engineering-demo/master/run.sh | \
    bash -s -- timescaledb

This command:

• downloads AcraServer, TimescaleDB, Prometheus, Grafana and PgAdmin images
• build metricsource image
• configures environment and starts demo stand using docker-compose

Demo stand based on TimescaleDB, which stores encrypted data. That data produced by metricsource container which connected to TimescaleDB through AcraServer.

At the initial stage, the TimescaleDB database will be filled with randomly generated metric data. Once started, the small daemon running in the metricsource container will continue to insert records into the database to simulate real processes.

Grafana also connected through AcraServer to TimescaleDB and can get decrypted data to build Temperature (demo data) graph.

Prometheus collects real metrics from AcraServer and show it on the dashboard AcraServer (real data).

Grafana shows dashboard with metrics collected from AcraServer and Prometheus.

1. Log into web TimescaleDB interface http://localhost:8008 using user/password: test@test.test/test.

2. Go to the Servers > postgresql > databases > test > Schemas > public > Tables > versions and open context menu with right-click. Select View/Edit Data > All rows and now you can see content of the table. Fields device and unit_id are encrypted. So, the data is stored in an encrypted form, but it is transparent for the Grafana.

You can easily interact with TimescaleDB through AcraServer:

docker exec -it \
  -ePGSSLMODE='verify-full' \
  -ePGSSLROOTCERT='scripts/ca.crt' \
  -ePGSSLKEY='/scripts/acra-client.key' \
  -ePGSSLCERT='/scripts/acra-client.crt' \
  timescaledb-metricsource-1 \
  psql  'postgres://postgres:test@acra-server:9393/test'

or directly:

docker exec -it -u postgres timescaledb-timescaledb-1 \
  psql test

1. TimescaleDB - connect to the database using the admin account postgres/test: postgresql://$HOST:5432.

2. pgAdmin - connect directly to the database using WebUI and user account login:test@test.test/password:test: http://localhost:8008

3. Grafana – see the dashboards with Acra metrics: http://localhost:3000.

4. Prometheus – examine the collected metrics: http://localhost:9090.

5. AcraServer – send some data directly through AcraServer: tcp://localhost:9393.

Python client application, transparent encryption/decryption/masking/tokenization with zones support, AcraServer, MySQL database.

curl https://raw.githubusercontent.com/cossacklabs/acra-engineering-demo/master/run.sh | \
    bash -s -- python-mysql

This command downloads a simple Python application that stores the data in a database, Acra Docker containers, MySQL database, sets up the environment, configures python application to connect to Acra, and provides a list of links for you to try.

The client application is a simple python console application that works with a database. The application talks with the database via Acra, Acra encrypts the data before sending it to a database, and decrypts the data when the app reads it from the database. Same it does transparently with tokenized data.

docker exec -it python-mysql_python_1 \
  python3 extended_example_with_zone.py --host=acra-server --port=9393 --generate_zone

$:
zone_id: DDDDDDDDPfBoWiixeMTUuEOk
zone public key in base64: b"UEC2\x00\x00\x00-\xbf\xc4\xd4\xa5\x02\x13ZTsg\x13\x88%R\xb5\x00\xc2\xbc\xe9\x9d\xa5\xa3i';\x7f)\xa8a\x9c\xdc\x9b\xc4\xba8\xb6\x04"

Call the extended_example_with_zone.py to encrypt the "top secret data" with a specific Zone. The application generates Zones using AcraServer HTTP API, then it uses Zone public key and Zone ID for encryption.

Set ZoneID DDDDDDDDPfBoWiixeMTUuEOk instead of existing in ./acra/examples/python/encryptor_config_with_zone.yaml:

sed -i 's/DDDDDDDDHHNqiSYFXkpxopYZ/DDDDDDDDPfBoWiixeMTUuEOk/g' ./acra/examples/python/extended_encryptor_config_with_zone.yaml

docker restart python-mysql_acra-server_1

Script reads data from data.json where stored array of entries as data examples.

docker exec -it python-mysql_python_1 \
  python3 extended_example_without_zone.py --host=acra-server --port=9393 --data=data.json

$:
data: [{'token_i32': 1234, 'token_i64': 645664, 'token_str': '078-05-1111', 'token_bytes': 'byt13es', 'token_email': 'john_wed@cl.com', 'data': 'John Wed, Senior Relationshop Manager', 'masking': '$112000', 'searchable': 'john_wed@cl.com'}, {'token_i32': 1235, 'token_i64': 645665, 'token_str': '078-05-1112', 'token_bytes': 'byt13es2', 'token_email': 'april_cassini@cl.com', 'data': 'April Cassini, Marketing Manager', 'masking': '$168000', 'searchable': 'april_cassini@cl.com'}, {'token_i32': 1236, 'token_i64': 645667, 'token_str': '078-05-1117', 'token_bytes': 'byt13es3', 'token_email': 'george_clooney@cl.com', 'data': 'George Clooney, Famous Actor', 'masking': '$780000', 'searchable': 'george_clooney@cl.com'}]

Read the data using the same ZoneId. AcraServer decrypts the data and returns plaintext:

docker exec -it python-mysql_python_1 \
  python3 extended_example_with_zone.py --host=acra-server --port=9393 --print  --zone_id=DDDDDDDDPfBoWiixeMTUuEOk
  
$:
Fetch data by query {}
 SELECT test.id, 'DDDDDDDDPfBoWiixeMTUuEOk' AS anon_1, test.data, test.masking, test.token_i32, test.token_i64, test.token_str, test.token_bytes, test.token_email 
FROM test
3
id  - zone_id - data - masking - token_i32 - token_i64 - token_str - token_bytes - token_email
1   - DDDDDDDDPfBoWiixeMTUuEOk - John Wed, Senior Relationshop Manager - $112000 - 1234 - 645664 - 078-05-1111 - byt13es - john_wed@cl.com
2   - DDDDDDDDPfBoWiixeMTUuEOk - April Cassini, Marketing Manager - $168000 - 1235 - 645665 - 078-05-1112 - byt13es2 - april_cassini@cl.com
3   - DDDDDDDDPfBoWiixeMTUuEOk - George Clooney, Famous Actor - $780000 - 1236 - 645667 - 078-05-1117 - byt13es3 - george_clooney@cl.com

To make sure that the data is stored in an encrypted form, read it directly from the database. Use --port=3306 and --host=mysql:

docker exec -it python-mysql_python_1 \
  python3 extended_example_with_zone.py --host=mysql --port=3306 --print --zone_id=DDDDDDDDPfBoWiixeMTUuEOk

$:
Fetch data by query {}
 SELECT test.id, 'DDDDDDDDYuWpBtCtrjpHyHta' AS anon_1, test.data, test.masking, test.token_i32, test.token_i64, test.token_str, test.token_bytes, test.token_email 
FROM test
3
id  - zone_id - data - masking - token_i32 - token_i64 - token_str - token_bytes - token_email
1   - DDDDDDDDYuWpBtCtrjpHyHta - %%%""""L@'/e_>I6躁iH 1Rz#X.5@@f+hRgjp�$~@oxշ '&T@ - -560604022 - -3330418728144437366 - ccxIvgP0iLp - ?a - 98LK9@sigT2.net
...

As expected, data and masking looks encrypted and token_* replaced with random values.

1. Log into web MySQL phpmyadmin interface http://localhost:8080.

2. Find the table and the data rows.

3. Compare data in result table and source json. All entries except id were encrypted or tokenized.

So, the data are protected and it is transparent for the Python application.

Usage of Zones provides compartmentalisation as different users of the same app will have different encryption keys. However, it's possible to use AcraServer without Zones.

1. To disable Zones open python-mysql/acra-server-config/acra-server.yaml and change zonemode_enable: true value to false, encryptor_config_file: "/config/extended_encryptor_config_with_zone.yaml" to encryptor_config_file: "/config/extended_encryptor_config_without_zone.yaml" and restart acra-server.

2. Write and read the data:

docker restart python-mysql_acra-server_1

docker exec -it python-mysql_python_1 \
  python3 extended_example_without_zone.py --host=acra-server --port=9393 --data=data.json

$:
data: [{'token_i32': 1234, 'token_i64': 645664, 'token_str': '078-05-1111', 'token_bytes': 'byt13es', 'token_email': 'john_wed@cl.com', 'data': 'John Wed, Senior Relationshop Manager', 'masking': '$112000', 'searchable': 'john_wed@cl.com'}, {'token_i32': 1235, 'token_i64': 645665, 'token_str': '078-05-1112', 'token_bytes': 'byt13es2', 'token_email': 'april_cassini@cl.com', 'data': 'April Cassini, Marketing Manager', 'masking': '$168000', 'searchable': 'april_cassini@cl.com'}, {'token_i32': 1236, 'token_i64': 645667, 'token_str': '078-05-1117', 'token_bytes': 'byt13es3', 'token_email': 'george_clooney@cl.com', 'data': 'George Clooney, Famous Actor', 'masking': '$780000', 'searchable': 'george_clooney@cl.com'}]


docker exec -it python-mysql_python_1 \
  python3 extended_example_without_zone.py --host=acra-server --port=9393 --print

$:
Fetch data by query {}
 SELECT test.id, test.data, test.masking, test.token_i32, test.token_i64, test.token_str, test.token_bytes, test.token_email 
FROM test
6
id  - data - masking - token_i32 - token_i64 - token_str - token_bytes - token_email
...
 - yMLDOzoMx@4juJOQbj.de78xxxx - -63551493 - -5945288817374683 - tk59cg2klQ7 - l҄
4   - John Wed, Senior Relationshop Manager - $112000 - 1234 - 645664 - 078-05-1111 - byt13es - john_wed@cl.com
5   - April Cassini, Marketing Manager - $168000 - 1235 - 645665 - 078-05-1112 - byt13es2 - april_cassini@cl.com
6   - George Clooney, Famous Actor - $780000 - 1236 - 645667 - 078-05-1117 - byt13es3 - george_clooney@cl.com
bash-5.1# python3 extended_example_without_zone.py --host=acra-server --port=9393 --print

Note: First 3 entries are encrypted and tokenized from previous example with zones and last 3 are new one encrypted in mode without zones.

1. MySQL – connect directly to the database using the admin account test/test: mysql://localhost:3306.

2. phpmyadmin - connect directly to the database using WebUI : http://localhost:8080

3. Prometheus – examine the collected metrics: http://localhost:9090.

4. Grafana – see the dashboards with Acra metrics: http://localhost:3000.

5. Jaeger – view traces: http://localhost:16686.

6. Docker-compose.python-mysql.yml file – read details about configuration and containers used in this example.

Take a look at the complete code of extended_example_with_zone.py and extended_example_without_zone.py.

Let's see how many code lines are necessary to encrypt some data using Acra. We will look at the example with Zones because it's more complicated and requires additional API call to fetch Zone.

1. The app gets a Zone using AcraServer API:

def get_zone():
    response = urlopen('{}/getNewZone'.format(ACRA_SERVER_API_ADDRESS))
    json_data = response.read().decode('utf-8')
    zone_data = json.loads(json_data)
    return zone_data['id'], b64decode(zone_data['public_key'])

2. The app reads JSON data and writes the data to the database as usual:

def write_data(data, connection):
   # here we encrypt our data and wrap into AcraStruct
   with open(data, 'r') as f:
      data = json.load(f)
   print("data: {}".format(data))
   rows = data
   if isinstance(data, dict):
      rows = [data]
   for row in rows:
      for k in ('data_str', 'data_i64', 'data_i32', 'email', 'token_bytes', 'masking'):
         row[k] = row[k].encode('ascii')
      connection.execute(
         test_table.insert(), row)

3. Nothing changes when reading the data from the database:

def print_data(connection, zone_id, columns):
   ...
   print("Fetch data by query {}\n",
         query.compile(dialect=postgresql.dialect(), compile_kwargs={"literal_binds": True}))
   result = connection.execute(query)
   result = result.fetchall()
   ...

Note: We skipped code related to output formatting.

These are all the code changes! 🎉

Django web application with client-side encryption (AcraWriter), decryption on AcraServer, PostgreSQL

Follow Integrating AcraServer into infrastructure guide.

Client-side encryption and Acra-side decryption: data is encrypted on the application side and decrypted on the AcraServer:

curl https://raw.githubusercontent.com/cossacklabs/acra-engineering-demo/master/run.sh | \
    bash -s -- django

This command downloads the code of Django website example, Acra Docker containers, PostgreSQL database, Prometheus, Grafana, pgAdmin images and sets up the environment, configures Django application to encrypt data, and provides a list of links for you to try.

The client application is the famous Django app example – the source code of djangoproject.com. We've updated their source code to protect blog posts. Application stores blog posts in PosgtreSQL database. We encrypt blog posts' content before storing in database, and decrypt when reading from database.

Django app encrypts the sensitive fields of blog posts into separate AcraStructs (author name, author email, content are encrypted; blog post ID and title are in plaintext).

Django app writes AcraStructs to the database and reads the decrypted posts through AcraServer (which pretends to be a database).

From the users' perspective, the website works as it used to. However, the blog posts are protected now.

Please add a temporary entry to the hosts file:

echo "$SERVER_IP www.djangoproject.example" >> /etc/hosts

where SERVER_IP is the IP address of the server that is running the Acra Engineering Demo (if you run the demo on your machine, set it to 127.0.0.1). Updating the hosts file is required because we will run the protected djangoproject site locally. You can remove this line when you stop needed to access the demo site.

1. Log into admin cabinet http://www.djangoproject.example:8000/admin/blog/entry/ using user/password: admin/admin. Add a blog post to the Blogs/Entries:

2. Open the blog posts' feed http://www.djangoproject.example:8000/weblog/ and see your fresh post.

Everything worked well! Now, let's check the content of the database.

Log into the web PostgreSQL interface http://www.djangoproject.example:8008 using user/password: test@test.test/test. Find your blog post in Servers > postgresql > databases > djangoproject > Schemas > public > Tables > blog_entries and open context menu with right-click. Select View/Edit Data > All rows and now you can see content of the table. Download and read the content – it's encrypted.

So, the blog posts are stored encrypted, but it's transparent for site visitors and admins.

Open Grafana dashboards to see the performance stats of AcraServer. We collect following metrics: the number of decrypted cryptographic containers (AcraStructs and AcraBlocks), request and response processing time.

Grafana is available at http://localhost:3000.

AcraServer can export detailed traces to Jaeger. Use this data to optimize the performance of the entire system.

Jaeger is available at http://localhost:16686.

There's more to explore:

1. PostgreSQL – connect directly to the database using the admin account postgres/test: postgresql://localhost:5432.

2. pgAdmin - connect directly to the database using WebUI and user account login:test@test.test/password:test: http://localhost:8008

3. Prometheus – examine the collected metrics: http://localhost:9090.

4. Grafana – see the dashboards with Acra metrics: http://localhost:3000.

5. Jaeger – view traces: http://localhost:16686.

6. Docker-compose.django.yml file – read details about configuration and containers used in this example.

So, was it easy to integrate Acra into Django application? Sure it was!

You can compare our repo to the original repo and see how few changes we introduced:

1. We've added Acra storage public key (L278) necessary for AcraWriter to encrypt the data:

ACRA_SERVER_PUBLIC_KEY = b64decode(SECRETS.get('acra_storage_public_key'))

2. We added AcraWriter as a dependency and wrapped the original fields with it:

import acrawriter.django

summary = acrawriter.django.TextField()
summary_html = acrawriter.django.TextField()
body = acrawriter.django.TextField()
body_html = acrawriter.django.TextField()
author = acrawriter.django.CharField(max_length=100)

3. We've also run a database migration that changed the fields' format from string to binary to store the encrypted data.

Those are all the code changes! 🎉

Python client application, client-side encryption with zones support, AcraServer, PostgreSQL database.

curl https://raw.githubusercontent.com/cossacklabs/acra-engineering-demo/master/run.sh | \
    bash -s -- python

This command downloads a simple Python application that stores the data in a database, Acra Docker containers, PostgreSQL database, sets up the environment, configures python application to encrypt data, and provides a list of links for you to try.

The client application is a simple python console application that works with a database. The application encrypts the data in AcraStructs before sending it to a database. The application reads the decrypted data through AcraServer (that are transparent for the application).

docker exec -it python_python_1 \
  python /app/example_with_zone.py --data="top secret data"

$:
data: top secret data
zone: DDDDDDDDFidFDxORlrleaUrC

Call the example_with_zone.py to encrypt the "top secret data" with a specific Zone. The application generates Zones using AcraServer HTTP API, then it uses Zone public key and Zone Id for encryption.

Read the data using the same ZoneId. AcraServer decrypts the data and returns plaintext:

docker exec -it python_python_1 \
  python /app/example_with_zone.py --print --zone_id=DDDDDDDDFidFDxORlrleaUrC

$:
use zone_id:  DDDDDDDDFidFDxORlrleaUrC
id  - zone - data - raw_data
1   - DDDDDDDDFidFDxORlrleaUrC - top secret data - top secret data

The output contains Zone Id, the decrypted data, and raw_data (stored in plaintext for the demo purposes),

Read the data using an invalid ZoneId. AcraServer will not decrypt the data and return the default data from /app/encryptor_config_with_zone.yaml config:

docker exec -it python_python_1 \
  python /app/example_with_zone.py --print --zone_id=DDDDDDDDFidFDxORlrleaU

$:
use zone_id:  DDDDDDDDEEaNtNHSbdOISJM
id  - zone - data - raw_data
1   - DDDDDDDDEEaNtNHSbdOISJM - test-data - top secret data

To make sure that the data is stored in an encrypted form, read it directly from the database:

docker exec -it python_python_1 \
  python /app/example_with_zone.py --print --zone_id=DDDDDDDDFidFDxORlrleaUrC --host=postgresql --port=5432

$:
use zone_id:  DDDDDDDDkOGnRsCBZEwXnHlL
id  - zone - data - raw_data
1   - DDDDDDDDkOGnRsCBZEwXnHlL - """"""""UEC2-CVs-K)'9@gJ-0 '&T@ {W|SҡϛڱY+:uKn"3Wɕ|Ict'JGCW;@ ̛W]aPI|Z ~*vI] - top secret data

As expected, no entity decrypts the data. The raw_data is stored as plaintext so nothing changes.

1. Log into web PostgreSQL interface http://localhost:8008 using user/password: test@test.test/test.

2. Find the table and the data rows.

3. Try reading the content of data field – it's encrypted!

So, the data is stored in an encrypted form, but it is transparent for the Python application.

Usage of Zones provides compartmentalisation as different users of the same app will have different encryption keys. However, it's possible to use AcraServer without Zones.

1. To disable Zones open python/acra-server-config/acra-server.yaml and change zonemode_enable: true value to false.

2. To disable Zones open python/acra-server-config/acra-server.yaml and change zonemode_enable: true value to false, encryptor_config_file: "/app/encryptor_config_with_zone.yaml" to encryptor_config_file: "/app/encryptor_config_without_zone.yaml and restart acra-server.

3. Write and read the data:

docker restart python_acra-server_1

docker exec -it python_python_1 \
  bash -c 'export EXAMPLE_PUBLIC_KEY="$(cat /pub_key_name.txt)" && \
  python /app/example_without_zone.py --data="secret data without zones"'

$:
insert data: secret data without zones

docker exec -it python_python_1 \
  bash -c 'export EXAMPLE_PUBLIC_KEY="$(cat /pub_key_name.txt)" && \  
  python /app/example_without_zone.py --print'

$:
id  - data                 - raw_data
2   - secret data without zones - secret data without zones

Note: AcraServer decrypts either AcraStructs with Zones or without Zones at the same time. Sending different kinds of AcraStructs without changing the mode will lead to decryption errors.

1. PostgreSQL – connect directly to the database using the admin account postgres/test: postgresql://localhost:5432.

2. pgAdmin - connect directly to the database using WebUI and user account login:test@test.test/password:test: http://localhost:8008

3. Prometheus – examine the collected metrics: http://localhost:9090.

4. Grafana – see the dashboards with Acra metrics: http://localhost:3000.

5. Jaeger – view traces: http://localhost:16686.

6. Docker-compose.python.yml file – read details about configuration and containers used in this example.

Take a look at the complete code of example_with_zone.py and example_without_zone.py.

Let's see how many code lines are necessary to encrypt some data using Acra. We will look at the example with Zones because it's more complicated and requires additional API call to fetch Zone.

1. The app gets a Zone using AcraServer API:

def get_zone():
    response = urlopen('{}/getNewZone'.format(ACRA_SERVER_API_ADDRESS))
    json_data = response.read().decode('utf-8')
    zone_data = json.loads(json_data)
    return zone_data['id'], b64decode(zone_data['public_key'])

2. The app uses ZoneId and Zone public key to encrypt the data –

encrypted_data = create_acrastruct(
  data.encode('utf-8'), key, zone_id.encode('utf-8'))

and writes the data to the database as usual:

connection.execute(
    test_table.insert(), data=encrypted_data,
    zone_id=zone_id.encode('utf-8'),
    raw_data=data)

3. Nothing changes when reading the data from the database:

result = connection.execute(
    select([cast(zone_id.encode('utf-8'), BYTEA), test_table]))
result = result.fetchall()

ZONE_ID_INDEX = 0
for row in result:
    print(
        "{:<3} - {} - {} - {}\n".format(
        row['id'], row[ZONE_ID_INDEX].decode('utf-8'),
        row['data'].decode('utf-8', errors='ignore'), row['raw_data']))

These are all the code changes! 🎉

Ruby on Rails web application, client-side encryption, AcraServer, PostgreSQL database.

curl https://raw.githubusercontent.com/cossacklabs/acra-engineering-demo/master/run.sh | \
    bash -s -- rails

This command downloads a Rails application that stores data in a database, Acra Docker containers, PostgreSQL database, Prometheus, Grafana, pgAdmin. Additionally, downloads a memcached and Elastic as project dependencies, sets up the environment, and provides a list of links for you to try.

The client application is a rubygems.org fork that works with a database. The application encrypts the data in AcraStructs before sending it to a database. The application reads the decrypted data through AcraServer (that are transparent for the application).

1. Sign up with any fictional account at sign_up page of the app. That credentials will be used later when uploading gem.

2. Verify your fictional email on a special page for development purposes.

3. Sign in via gem CLI tool with credentials used in the step 2:

gem signin --host=http://www.rubygems.example:8000

4. Use already built gem rails/my-example-gem-0.1.0.gem or build your own:

bundle gem my-example-gem
cd my-example-gem/

cat > ./my-example-gem.gemspec <<'EOF'
lib = File.expand_path("../lib", __FILE__)
$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
require "my/example/gem/version"

Gem::Specification.new do |spec|
  spec.name          = "my-example-gem"
  spec.version       = My::Example::Gem::VERSION
  spec.authors       = ["My Example Name"]
  spec.email         = ["my@email.example"]

  spec.summary       = %q{Example Gem}
  spec.description   = %q{This is Example Gem}
  spec.homepage      = "http://site.example"
  spec.license       = "MIT"

  if spec.respond_to?(:metadata)
    spec.metadata["homepage_uri"] = spec.homepage
    spec.metadata["source_code_uri"] = "http://site.example"
    spec.metadata["changelog_uri"] = "http://site.example/CHANGELOG.md"
  else
    raise "RubyGems 2.0 or newer is required to protect against " \
      "public gem pushes."
  end

  # Specify which files should be added to the gem when it is released.
  # The `git ls-files -z` loads the files in the RubyGem that have been added into git.
  spec.files         = Dir.chdir(File.expand_path('..', __FILE__)) do
    `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
  end
  spec.bindir        = "exe"
  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
  spec.require_paths = ["lib"]

  spec.add_development_dependency "bundler", "~> 1.17"
  spec.add_development_dependency "rake", "~> 10.0"
  spec.add_development_dependency "rspec", "~> 3.0"
end
EOF

gem build my-example-gem

Now we are ready to push our gem to the application:

gem push my-example-gem-0.1.0.gem --host=http://www.rubygems.example:8000

After previous step we can see information about uploaded gem: http://www.rubygems.example:8000/gems/my-example-gem

1. Log into web PostgreSQL interface http://www.rubygems.example:8008 using user/password: test@test.test/test.

2. Go to the Servers > postgresql > databases > rubygems > Schemas > public > Tables > versions and open context menu with right-click. Select View/Edit Data > All rows and now you can see content of the table.

1. PostgreSQL – connect directly to the database using the admin account rubygems/rubygems: postgresql://www.rubygems.example:5432.

2. pgAdmin - connect directly to the database using WebUI and user account login:test@test.test/password:test: http://localhost:8008

3. Prometheus – examine the collected metrics: http://localhost:9090.

4. Grafana – see the dashboards with Acra metrics: http://localhost:3000.

5. Jaeger – view traces: http://localhost:16686.

6. AcraServer – send some data directly through AcraServer: tcp://www.rubygems.example:9393.

1. Add gem activerecord_acrawriter to Gemfile:

+gem 'activerecord_acrawriter'

2. Modify dependency.rb to encrypt data of unresolved_name column in Dependency model:

@@ -1,3 +1,5 @@
+require 'activerecord_acrawriter'
+
 class Dependency < ApplicationRecord
   belongs_to :rubygem
   belongs_to :version
@@ -11,6 +13,8 @@

   attr_accessor :gem_dependency

+  attribute :unresolved_name, AcraType.new
+
   def self.unresolved(rubygem)
     where(unresolved_name: nil, rubygem_id: rubygem.id)
   end

3. And finally modify version.rb encrypt data of authors, description and summary columns in Version model:

@@ -1,4 +1,5 @@
 require 'digest/sha2'
+require 'activerecord_acrawriter'

 class Version < ApplicationRecord
   belongs_to :rubygem, touch: true
@@ -21,7 +22,7 @@
   validate :authors_format, on: :create
   validate :metadata_links_format

-  class AuthorType < ActiveModel::Type::String
+  class AuthorType < AcraType
     def cast_value(value)
       if value.is_a?(Array)
         value.join(', ')
@@ -32,3 +33,5 @@
   end

   attribute :authors, AuthorType.new
+  attribute :description, AcraType.new
+  attribute :summary, AcraType.new

These are all the code changes! 🎉

Python client application, transparent encryption with zones support, AcraServer, CockroachDB database.

curl https://raw.githubusercontent.com/cossacklabs/acra-engineering-demo/master/run.sh | \
    bash -s -- cockroachdb

This command downloads a simple Python application that stores the data in a database, Acra Docker containers, CockroachDB database, sets up the environment, configures python application to connect to Acra, and provides a list of links for you to try.

The client application is a simple python console application that works with a database. The application talks with the database via Acra, Acra encrypts the data before sending it to a database, and decrypts the data when the app reads it from the database. Same it does transparently with tokenized data.

docker exec -it cockroachdb_python_1 \
  python3 extended_example_with_zone.py --host=acra-server --port=9393 --generate_zone

$:
zone_id: DDDDDDDDPfBoWiixeMTUuEOk
zone public key in base64: b"UEC2\x00\x00\x00-\xbf\xc4\xd4\xa5\x02\x13ZTsg\x13\x88%R\xb5\x00\xc2\xbc\xe9\x9d\xa5\xa3i';\x7f)\xa8a\x9c\xdc\x9b\xc4\xba8\xb6\x04"

Call the extended_example_with_zone.py to generate Zone using AcraServer HTTP API.

Set ZoneID DDDDDDDDPfBoWiixeMTUuEOk instead of existing in ./acra/examples/python/extended_encryptor_config_with_zone.yaml:

sed -i 's/DDDDDDDDHHNqiSYFXkpxopYZ/DDDDDDDDPfBoWiixeMTUuEOk/g' ./acra/examples/python/extended_encryptor_config_with_zone.yaml

docker restart cockroachdb_acra-server_1

Script reads data from data.json where stored array of entries as data examples.

docker exec -it cockroachdb_python_1 \
  python3 extended_example_without_zone.py --host=acra-server --port=9393 --data=data.json

$:
data: [{'token_i32': 1234, 'token_i64': 645664, 'token_str': '078-05-1111', 'token_bytes': 'byt13es', 'token_email': 'john_wed@cl.com', 'data': 'John Wed, Senior Relationshop Manager', 'masking': '$112000', 'searchable': 'john_wed@cl.com'}, {'token_i32': 1235, 'token_i64': 645665, 'token_str': '078-05-1112', 'token_bytes': 'byt13es2', 'token_email': 'april_cassini@cl.com', 'data': 'April Cassini, Marketing Manager', 'masking': '$168000', 'searchable': 'april_cassini@cl.com'}, {'token_i32': 1236, 'token_i64': 645667, 'token_str': '078-05-1117', 'token_bytes': 'byt13es3', 'token_email': 'george_clooney@cl.com', 'data': 'George Clooney, Famous Actor', 'masking': '$780000', 'searchable': 'george_clooney@cl.com'}]

Read the data using the same ZoneId. AcraServer decrypts the data and returns plaintext:

docker exec -it cockroachdb_python_1 \
  python3 extended_example_with_zone.py --host=acra-server --port=9393 --print  --zone_id=DDDDDDDDPfBoWiixeMTUuEOk
  
$:
Fetch data by query {}
 SELECT test.id, 'DDDDDDDDPfBoWiixeMTUuEOk' AS anon_1, test.data, test.masking, test.token_i32, test.token_i64, test.token_str, test.token_bytes, test.token_email 
FROM test
3
id  - zone_id - data - masking - token_i32 - token_i64 - token_str - token_bytes - token_email
1   - DDDDDDDDPfBoWiixeMTUuEOk - John Wed, Senior Relationshop Manager - $112000 - 1234 - 645664 - 078-05-1111 - byt13es - john_wed@cl.com
2   - DDDDDDDDPfBoWiixeMTUuEOk - April Cassini, Marketing Manager - $168000 - 1235 - 645665 - 078-05-1112 - byt13es2 - april_cassini@cl.com
3   - DDDDDDDDPfBoWiixeMTUuEOk - George Clooney, Famous Actor - $780000 - 1236 - 645667 - 078-05-1117 - byt13es3 - george_clooney@cl.com

To make sure that the data is stored in an encrypted form, read it directly from the database. Use --port=26257 and --host=roach1:

docker exec -it cockroachdb_python_1 \
  python3 extended_example_with_zone.py --host=roach1 --port=26257 --print --zone_id=DDDDDDDDPfBoWiixeMTUuEOk

$:
Fetch data by query {}
 SELECT test.id, 'DDDDDDDDYuWpBtCtrjpHyHta' AS anon_1, test.data, test.masking, test.token_i32, test.token_i64, test.token_str, test.token_bytes, test.token_email 
FROM test
3
id  - zone_id - data - masking - token_i32 - token_i64 - token_str - token_bytes - token_email
1   - DDDDDDDDYuWpBtCtrjpHyHta - %%%""""L@'/e_>I6躁iH 1Rz#X.5@@f+hRgjp�$~@oxշ '&T@ - -560604022 - -3330418728144437366 - ccxIvgP0iLp - ?a - 98LK9@sigT2.net
...

1. CockroachDB – connect directly to the database using the user root and DB defaultdb: postgresql://localhost:26257.

2. Prometheus – examine the collected metrics: http://localhost:9090.

3. Grafana – see the dashboards with Acra metrics: http://localhost:3000.

4. Jaeger – view traces: http://localhost:16686.

5. Docker-compose.python.yml file – read details about configuration and containers used in this example.

Learn how to configure AcraCensor – SQL firewall – to allow or deny specific queries and make your application more steady against SQL injections.

What's inside: OWASP Mutillidae vulnerable web application, AcraServer, AcraCensor.

Follow the guide: Acra firewall example.

Learn how to build high availability and balanced infrastructures for AcraServer based on HAProxy.

What's inside: python client application, AcraServer, HAProxy.

Follow the guide: Acra load balancing example.

Learn how to use searchable encryption and search through encrypted data without decryption.

$ curl https://raw.githubusercontent.com/cossacklabs/acra-engineering-demo/master/run.sh | \
    bash -s -- python-searchable

This command downloads a simple Python application that stores the data in a database, Acra Docker containers, MySQL and PostgreSQL databases, sets up the environment, configures python application to connect to Acra, and provides a list of links for you to try.

The client application is a simple python console application that works with a database. The application talks with the database via Acra, Acra encrypts the data before sending it to a database (either PostgreSQL or MySQL), and decrypts the data when the app reads it from the database.

By default, Acra is configured to use PostgreSQL as a database. But you can make it use MySQL. To do that, open ./python-searchable/acra-server-config/acra-server.yaml and uncomment appropriate config lines. After, restart the AcraServer:

$ docker restart python-searchable-acra-server-1

Also, the client requires an appropriate flag to know which driver to use (--postgresql or --mysql). To simplify the workflow, export this flag into the DB variable:

$ export DB="--postgresql"

or

$ export DB="--mysql"

To insert data, run:

$ docker exec -it python-searchable-python-1 \
    python /app/searchable.py $DB \
    --data=searchable.json

data:
[{'searchable_email': 'john_wed@cl.com', 'searchable_name': 'John'},
 {'searchable_email': 'april_cassini@cl.com', 'searchable_name': 'April'},
 {'searchable_email': 'george_clooney@cl.com', 'searchable_name': 'George'}]

The client reads rows from searchable.json file. You can find it at ./acra/examples/python/searchable.json.

Select all rows with the following command:

$ docker exec -it python-searchable-python-1 \
    python /app/searchable.py $DB \
    --print

Fetch data by query SELECT test.id, test.searchable_name, test.searchable_email 
FROM test
- id: 1
  searchable_name: John
  searchable_email: john_wed@cl.com
- id: 2
  searchable_name: April
  searchable_email: april_cassini@cl.com
- id: 3
  searchable_name: George
  searchable_email: george_clooney@cl.com

TOTAL 3

The rows are fetched from the database and decrypted by the AcraServer.

The database's schema has two columns: searchable_name and searchable_email. The AcraServer is configured to support both for searchable encryption. For example, let's search for the name April:

$ docker exec -it python-searchable-python-1 \
    python /app/searchable.py $DB \
    --print \
    --search_name 'April'
  
Fetch data by query SELECT test.id, test.searchable_name, test.searchable_email 
FROM test 
WHERE test.searchable_name = 'April'
- id: 2
  searchable_name: April
  searchable_email: april_cassini@cl.com

TOTAL 1

Let's also search for the email john_wed@cl.com:

$ docker exec -it python-searchable-python-1 \
    python /app/searchable.py $DB \
    --print \
    --search_email 'john_wed@cl.com'

Fetch data by query SELECT test.id, test.searchable_name, test.searchable_email 
FROM test 
WHERE test.searchable_email = 'john_wed@cl.com'
- id: 1
  searchable_name: John
  searchable_email: john_wed@cl.com

TOTAL 1

Searchable encryption supports only the exact match, therefore the search for john instead of John results in no rows:

$ docker exec -it python-searchable-python-1 \
    python /app/searchable.py $DB \
    --print \
    --search_name 'john'

Fetch data by query SELECT test.id, test.searchable_name, test.searchable_email 
FROM test 
WHERE test.searchable_name = 'john'

TOTAL 0

To make sure that the data is stored in an encrypted form, read it directly from the database.

If you are using PostgreSQL, run:

$ docker exec -it python-searchable-python-1 \
    python /app/searchable.py \
    --postgresql \
    --host=postgresql \
    --port=5432 \
    --print

Or, if you are using MySQL:

$ docker exec -it python-searchable-python-1 \
    python /app/searchable.py \
    --mysql \
    --host=mysql \
    --port=3306 \
    --print

In both cases you will see a gibberish on the screen:

Fetch data by query SELECT test.id, test.searchable_name, test.searchable_email 
FROM test
- id: 1
  searchable_name: NV9q.kq;gi;VB%%%""""L@
                                           
                                           D"Lk=|	"{{VnίڶHZozj#[U
,=$F	 @
          zLA3td0zi

  searchable_email: lh_|Μ-\W{IЇ%%%""""L@
                                         -7:&}Ef|9VbNVӁ
                                                       |S+|Rn4@
iU(C9@A0bklXvyڠ-                                               2+IgGH
- id: 2
  searchable_name: &}a7-H*YÚ%%%""""L@
                                      CaC/D1_\CR
                                                ز&EAMN*)忪;X?Q>8	-jM5w@
                                                                              V&pmJp)=+Ն!~
  searchable_email: ^iJ E͐Rv@dNsF%%%""""L@
                                          KZCz8xՈDP8 uLFlW@
                                                           xFɄlAMI_9Rz)
                                                                       �Hf29A
- id: 3
  searchable_name: %:xyo{EA|ǹe51d3^?%%%""""L@
                                              򢑆* du!=iJnqKs@ƅHiцyN!@[5إh@
                                                                         &xضA=	Djbʇ7>Fxʪ,3pe
  searchable_email: :1.&ڷmBA;DF/ؼAP%%%""""L@
                                             wJq"]gǡ1ib.iՃBլJԪgj[;2h+_v@
                                                                        X

Pm2W¤8
      ?@

TOTAL 3

1. Acra encryptor config - study the Acra configuration that tells how to encrypt and index data.

2. Prometheus – examine the collected metrics: http://localhost:9090.

3. Grafana – see the dashboards with Acra metrics: http://localhost:3000.

4. Jaeger – view traces: http://localhost:16686.

1. PostgreSQL – connect directly to the database using the admin account test/test: postgresql://localhost:5432.

2. pgAdmin - connect directly to the database using WebUI and user account login:test@test.test/password:test: http://localhost:8008. While connecting to the server, don't forget to change the admin user to test instead of postgresql.

1. MySQL – connect directly to the database using the admin account test/test: mysql://localhost:3306.

2. phpmyadmin - connect directly to the database using WebUI : http://localhost:8080

1. docker-compose.python-searchable.yml file – read details about configuration and containers used in this example.

Take a loot at the complete code of searchable.py.

Let's see how many code lines are necessary to encrypt and search some data using Acra.

1. The app reads JSON data and writes it to the database as usual:

    with open(data, 'r') as f:
        data = json.load(f)
    to_escape = ('searchable_name', 'searchable_email')
   
    for row in rows:
        for k in row:
            if k in to_escape:
                row[k] = row[k].encode('ascii')
        connection.execute(table.insert(), row)

2. Nothing changes when reading the data from the database:

   query = select(table_columns)
   # ...
   rows = connection.execute(query).fetchall()

3. To search for some values, regular WHERE clause is used:

   query = select(table_columns)
   
   
    if search_email:
        search_email = search_email.encode('ascii')
        query = query.where(table.c.searchable_email == search_email)
   
    if search_name:
        search_name = search_name.encode('ascii')
        query = query.where(table.c.searchable_name == search_name)
   
   rows = connection.execute(query).fetchall()

NOTE: We skipped code related to output formatting.

These are all the code changes! 🎉

Let us know if you have any questions by dropping an email to dev@cossacklabs.com.

1. Acra website – learn about all Acra features, defense in depth, how it's better than "just TLS" and available licenses.
2. Acra Community Edition – Acra Community Edition repository.
3. Acra docs – all Acra docs and guides.

Need help in configuring Acra? Read more about support options and Acra Enterprise Edition.