Binary Deployment Guide

Prerequisites

• Unix-based operating system (Linux or macOS)
• Either curl OR wget for downloading files
• unzip for extracting files
• sudo privileges for installing and configuring services
• At least 1GB of available RAM
• 1GB of free disk space
• 2 CPU cores minimum (3+ cores recommended)

Quick Start

One-Line Installation

Using curl:

# Install latest version
curl -fsSL https://dbconvert.nyc3.digitaloceanspaces.com/downloads/streams/latest/binary-install.sh | sh

# Install specific version
curl -fsSL https://dbconvert.nyc3.digitaloceanspaces.com/downloads/streams/latest/binary-install.sh | sh -s -- --version v1.0.0

Using wget:

# Install latest version
wget -qO- https://dbconvert.nyc3.digitaloceanspaces.com/downloads/streams/latest/binary-install.sh | sh

# Install specific version
wget -qO- https://dbconvert.nyc3.digitaloceanspaces.com/downloads/streams/latest/binary-install.sh | sh -s -- --version v1.0.0

After installation completes:

cd ~/dbconvert-streams-binary && ./start.sh

System Components

Core Services

1. stream-api (port 8020)

   • REST API for managing database connections and stream configurations
   • Handles authentication and authorization
   • Integrates with Consul for configuration storage

2. stream-reader

   • Reads from source databases
   • Publishes to NATS
   • Reports metrics and status

3. stream-writer

   • Consumes data from NATS
   • Writes to target databases
   • Handles schema creation

Infrastructure Services

• NATS: Message streaming
• Hashicorp Vault: Secrets management for securely storing:
  • Database passwords and credentials
  • SSL/TLS certificates
  • Client certificates
  • Other sensitive connection information
• Consul: Service discovery, health monitoring, and persistent storage for stream and connection configurations

Service Management

Starting Services

cd ~/dbconvert-streams-binary && ./start.sh

Stopping Services

cd ~/dbconvert-streams-binary && ./stop.sh

Manual Service Control

sudo systemctl start/stop nats
sudo systemctl start/stop consul
sudo systemctl start/stop vault
sudo systemctl start/stop stream-api
sudo systemctl start/stop stream-reader
sudo systemctl start/stop stream-writer
sudo systemctl start/stop nginx

Configuration Files

Core Configuration Files

• nats.conf: NATS data directory settings, connection parameters, and JetStream configuration
• vault.config.hcl: Vault storage backend settings, listener configuration, and security parameters
• consul.config.hcl: Consul configuration including data directory and server settings

Systemd Service Files

• nats.service: NATS service configuration
• vault.service: Vault service configuration
• consul.service: Consul service configuration
• stream-api.service: API service configuration
• stream-reader.service: Reader service configuration
• stream-writer.service: Writer service configuration

Logging

All logs are stored in ~/dbconvert-streams-binary/logs/ by default:

• api.log: API service logs
• reader.log: Source reader logs
• writer.log: Target writer logs
• nats.log: NATS server logs
• vault.log: Vault server logs
• consul.log: Consul server logs

Logging Configuration

DBConvert Streams services support the following logging flags for customized logging:

--log-file

Specifies a custom location and filename for the service's log file:

./stream-api --log-file=../logs/api.log
./stream-reader --log-file=../logs/source.log 
./stream-writer --log-file=../logs/target.log

--log-level

Sets the logging verbosity level. Default is info (includes info, warning, and error messages). Use debug for more verbose output:

./stream-api --log-level=debug
./stream-reader --log-level=debug
./stream-writer --log-level=debug

Updating

To update DBConvert Streams:

cd ~/dbconvert-streams-binary && ./binary-install.sh

Uninstallation

To remove DBConvert Streams and dependencies:

cd ~/dbconvert-streams-binary && ./uninstall.sh

The uninstall script will:

• Stop all services
• Remove systemd service files
• Delete configuration directories
• Remove binary files
• Clean up data directories

Troubleshooting

1. Service Won't Start

   • Check logs in ~/dbconvert-streams-binary/logs/
   • Verify service configuration
   • Ensure required ports are available

2. Permission Issues

   • Verify sudo access
   • Check file permissions
   • Review service user settings

3. Communication Problems

   • Confirm NATS is running
   • Check network connectivity
   • Verify service discovery

For additional support, contact us at https://streams.dbconvert.com/contact