Start a Cluster in Docker (Insecure)

On this page

Once you've installed the official CockroachDB Docker image, it's simple to run an insecure multi-node cluster across multiple Docker containers on a single host, using Docker volumes to persist node data.

Before you begin

• Make sure you have already installed the official CockroachDB Docker image.
• For quick SQL testing or application development, consider running a single-node cluster. When you use Docker to run a single-node cluster, some additional features are available to assist you with testing and development. See Start a single-node cluster. Single-node clusters are not highly available or fault-tolerant, and are not appropriate for production use.
• Running multiple nodes on a single host is useful for testing CockroachDB, but it's not highly available or fault tolerant, and is not suitable for production. To run a physically-distributed cluster in containers, use an orchestration tool like Kubernetes. See Orchestration for more details, and review the Production Checklist.

Start a multi-node cluster

Step 1. Create a bridge network

Since you'll be running multiple Docker containers on a single host, with one CockroachDB node per container, you need to create what Docker refers to as a bridge network. The bridge network will enable the containers to communicate as a single cluster while keeping them isolated from external networks.

docker network create -d bridge roachnet

We've used roachnet as the network name here and in subsequent steps, but feel free to give your network any name you like.

Step 2: Create Docker volumes for each cluster node

Cockroach Labs recommends that you store cluster data in Docker volumes rather than in the storage layer of the running container. Otherwise, if a Docker container is inadvertently deleted, its data is inaccessible.

Create a Docker volume for each container:

docker volume create roach1

docker volume create roach2

docker volume create roach3

Step 3. Start the cluster

1. Start the first node:

   docker run -d \
   --name=roach1 \
   --hostname=roach1 \
   --net=roachnet \
   -p 26257:26257 -p 8080:8080  \
   -v "roach1:/cockroach/cockroach-data"  \
   cockroachdb/cockroach-unstable:v23.1.0-rc.1 start \
   --insecure \
   --join=roach1,roach2,roach3
   

2. This command creates a container and starts the first CockroachDB node inside it. Take a moment to understand each part:

   • docker run: The Docker command to start a new container.
   • -d: This flag runs the container in the background so you can continue the next steps in the same shell.
   • --name: The name for the container. This is optional, but a custom name makes it significantly easier to reference the container in other commands, for example, when opening a Bash session in the container or stopping the container.
   • --hostname: The hostname for the container. You will use this to join other containers/nodes to the cluster.
   • --net: The bridge network for the container to join. See step 1 for more details.
   • -p 26257:26257 -p 8080:8080: These flags map the default port for inter-node and client-node communication (26257) and the default port for HTTP requests to the DB Console (8080) from the container to the host. This enables inter-container communication and makes it possible to call up the DB Console from a browser.
   • -v "roach1:/cockroach/cockroach-data": This flag mounts a host directory as a data volume. This means that data and logs for this node will be stored in the roach1 volume on the host and will persist after the container is stopped or deleted. For more details, see Docker's volumes topic.
   • cockroachdb/cockroach-unstable:v23.1.0-rc.1 start --insecure --join: The CockroachDB command to start a node in the container in insecure mode. The --join flag specifies the hostname of each node that will initially comprise your cluster. Otherwise, all cockroach start defaults are accepted. Note that since each node is in a unique container, using identical default ports won’t cause conflicts.

3. Start two more nodes:

   docker run -d \
   --name=roach2 \
   --hostname=roach2 \
   --net=roachnet \
   -v "roach2:/cockroach/cockroach-data" \
   cockroachdb/cockroach-unstable:v23.1.0-rc.1 start \
   --insecure \
   --join=roach1,roach2,roach3
   

   docker run -d \
   --name=roach3 \
   --hostname=roach3 \
   --net=roachnet \
   -v "roach3:/cockroach/cockroach-data" \
   cockroachdb/cockroach-unstable:v23.1.0-rc.1 start \
   --insecure \
   --join=roach1,roach2,roach3
   

4. Perform a one-time initialization of the cluster:

   docker exec -it roach1 ./cockroach init --insecure
   

   The following message displays:

   Cluster successfully initialized
   

   Each node also prints helpful startup details to its log. For example, the following command retrieves the start-up details for roach1.

   $ docker exec -it roach1 grep 'node starting' cockroach-data/logs/cockroach.log -A 11
   

   The output will look something like this:

   CockroachDB node starting at 
   build:               CCL v23.1.0-rc.1 @ 2023-05-02 00:00:00 (go1.19) (go1.12.6)
   webui:               http://roach1:8080
   sql:                 postgresql://root@roach1:26257?sslmode=disable
   client flags:        /cockroach/cockroach <client cmd> --host=roach1:26257 --insecure
   logs:                /cockroach/cockroach-data/logs
   temp dir:            /cockroach/cockroach-data/cockroach-temp273641911
   external I/O path:   /cockroach/cockroach-data/extern
   store[0]:            path=/cockroach/cockroach-data
   status:              initialized new cluster
   clusterID:           1a705c26-e337-4b09-95a6-6e5a819f9eec
   nodeID:              1
   

Step 4. Use the built-in SQL client

Now that your cluster is live, you can use any node as a SQL gateway. To test this out, let's use the docker exec command to start the built-in SQL shell in the roach1 container.

1. Start the SQL shell in the first container:

   $ docker exec -it roach1 ./cockroach sql --insecure
   

2. Run some basic CockroachDB SQL statements:

   > CREATE DATABASE bank;
   

   > CREATE TABLE bank.accounts (id INT PRIMARY KEY, balance DECIMAL);
   

   > INSERT INTO bank.accounts VALUES (1, 1000.50);
   

   > SELECT * FROM bank.accounts;
   

     id | balance
   +----+---------+
      1 | 1000.50
   (1 row)
   

3. Exit the SQL shell on roach1 and open a new shell on roach2:

   > \q
   

   $ docker exec -it roach2 ./cockroach sql --insecure
   

4. Run the same SELECT query as before:

   > SELECT * FROM bank.accounts;
   

     id | balance
   +----+---------+
      1 | 1000.50
   (1 row)
   

   As you can see, roach1 and roach2 perform identically as SQL gateways.

5. Exit the SQL shell on roach2:

   > \q
   

Step 5. Run a sample workload

CockroachDB also comes with a number of built-in workloads for simulating client traffic. Let's run the workload based on CockroachDB's sample vehicle-sharing application, MovR.

1. Load the initial dataset on roach1:

   $ docker exec -it roach1 ./cockroach workload init movr \
   'postgresql://root@roach1:26257?sslmode=disable'
   

2. Run the workload for five minutes:

   $ docker exec -it roach1 ./cockroach workload run movr \
   --duration=5m \
   'postgresql://root@roach1:26257?sslmode=disable'
   

Step 6. Access the DB Console

The DB Console gives you insight into the overall health of your cluster as well as the performance of the client workload.

1. When you started the first node's container, you mapped the node's default HTTP port 8080 to port 8080 on the Docker host, so go to http://localhost:8080.

2. On the Cluster Overview, notice that three nodes are live, with an identical replica count on each node:

   

   This demonstrates CockroachDB's automated replication of data via the Raft consensus protocol.

   Note:

   Capacity metrics can be incorrect when running multiple nodes on a single machine. For more details, see this limitation.

3. Click Metrics to access a variety of time series dashboards, including graphs of SQL queries and service latency over time:

   

4. Use the Databases, Statements, and Jobs pages to view details about your databases and tables, to assess the performance of specific queries, and to monitor the status of long-running operations like schema changes, respectively.

Step 7. Stop the cluster

1. Use the docker stop and docker rm commands to stop and remove the containers (and therefore the cluster):

   $ docker stop roach1 roach2 roach3
   

   $ docker rm roach1 roach2 roach3
   

2. If you do not plan to restart the cluster, you can also remove the Docker volumes:

   docker volume rm roach1 roach2 roach3
   

Start a single-node cluster

When you use the cockroach start-single-node command to start a single-node cluster with Docker, additional features are available to help with testing and development.

• You can optionally set the following Docker environment variables to create a database and user automatically and to set a password for the user.

  • COCKROACH_DATABASE
  • COCKROACH_USER
  • COCKROACH_PASSWORD

  To prevent loss of a cluster's existing data, the environment variables are used only if the /cockroach/cockroach-data directory within the container is empty.

• You can optionally mount a directory of initialization scripts into the docker-entrypoint-initdb.d directory within the container. These scripts are run after CockroachDB starts and after the database and user (if specified as environment variables) have been created. The scripts run in the alphanumeric sort order imposed by your locale. The init scripts are run only if the /cockroach/cockroach-data directory within the container is empty.

During local development and testing, you can re-initialize the default database, user, and password by deleting the contents of /cockroach/cockroach-data within the running container and then restarting the container.

This section shows how to start a single-node cluster that uses these features.

Step 1. Create a Docker volume for the node

Cockroach Labs recommends that you store cluster data in a Docker volume rather than in the storage layer of the running container. Otherwise, if a Docker container is inadvertently deleted, its data is inaccessible.

To create the Docker volume where the cluster will store its data, run the following:

docker volume create roach-single

Step 2. Start the cluster

This section shows how to start a single-node cluster that:

• Stores its data in the roach-single volume on the Docker host, which is mounted on the /cockroach/cockroach-data directory within the container.

• If the /cockroach/cockroach-data directory within the container is empty, creates the specified database, user, and password automatically.

  Tip:

  Instead of specifying each value directly by using the -e or --env flag, you can store them in a file on the Docker host. Use one key-value pair per line and set the --env-file flag to the file's path.

• Bind-mounts the ~/init-scripts directory on the Docker host onto the /docker-entrypoint-initdb.d directory within the container. Initialization scripts stored in this directory are run after CockroachDB starts and the default database, user, and password are initialized.

• Accepts database client connections on hostname roach-single on port 26257.

• Accepts connections to the DB Console on hostname roach-single on port 8080.

The cockroach process listens on 127.0.0.1:26257 and localhost:26257, and this cannot be changed for single-node cluster running in a container. The --listen-address option is ignored.

1. Start the cluster node.

   docker run -d \
             --env COCKROACH_DATABASE={DATABASE_NAME} \
             --env COCKROACH_USER={USER_NAME} \
             --env COCKROACH_PASSWORD={PASSWORD} \
             --name=roach-single \
             -p 26257:26257 -p 8080:8080 \
             -v "roach-single:/cockroach/cockroach-data" \
             -v "~/init-scripts:/docker-entrypoint-initdb.d" \
             cockroachdb/cockroach:latest start-single-node
   

   By default, a certs directory is created and CockroachDB starts in secure mode.

   Note:

   The COCKROACH_DATABASE, COCKROACH_USER, and COCKROACH_PASSWORD environment variables and the contents of the /docker-entrypoint-initdb.d directory are ignored if you use cockroach start rather than cockroach start-single-node. They are also ignored if data exists in the /cockroach/cockroach-data directory within the container.

   Docker adds a DNS entry that resolves the hostname roach-single to the container's IP address in Docker's default network. The following examples use this hostname.

2. After the cluster is initialized, the cluster node prints helpful startup details to its log, including the DB Console URL and the SQL connection string. To retrieve roach-single's startup details:

   docker exec -it roach-single grep 'node starting' cockroach-data/logs/cockroach.log -A 11
   

   CockroachDB node starting at 2022-09-28 14:35:27.495508396 +0000 UTC m=+1.370085757 (took 0.5s)
   build:               CCL  @ 2022/09/26 18:49:07 (go1.19.1)
   webui:               https://172.18.0.3:8080
   sql:                 postgresql://root@172.18.0.3:26257/defaultdb?sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt
   sql (JDBC):          jdbc:postgresql://172.18.0.3:26257/defaultdb?sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt&user=root
   RPC client flags:    /cockroach/cockroach <client cmd> --host=172.18.0.3:26257 --certs-dir=certs
   logs:                /cockroach/cockroach-data/logs
   temp dir:            /cockroach/cockroach-data/cockroach-temp2611102055
   external I/O path:   /cockroach/cockroach-data/extern
   store[0]:            path=/cockroach/cockroach-data
   storage engine:      pebble
   clusterID:           60f29a4e-1c87-4b0c-805d-eb73460766b1
   status:              initialized new cluster
   nodeID:              1
   

Step 3. Connect to the cluster

1. After the cluster is initialized, you can connect to it, run tests on it, and stop it using the same instructions as a multi-node cluster. To monitor the cluster node's logs interactively:

   docker log roach-single --follow
   

   To stop monitoring the logs, press Ctrl+C to exit the docker log command.

2. To connect to the cluster interactively using the cockroach sql command-line interface, set --url cluster's SQL connection string, which is printed next to sql: in the cluster's startup details. You can replace the IP address with the hostname of the Docker container so that the script continues to work if the IP address changes. To run cockroach sql on the roach-single cluster:

   docker exec -it roach-single ./cockroach sql --url="postgresql://root@roach-single:26257/defaultdb?sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt"
   

3. To connect to the cluster using DB Console in your web browser, navigate to the address printed next to webui: in the cluster's startup details. You can replace the IP address with the hostname of the Docker container. To connect to the roach-single cluster, navigate to https://roach-single:8080/ for a secure cluster or http://roach-single:8080/ for an insecure cluster.

Step 4. Stop the cluster

1. Use the docker stop and docker rm commands to stop and remove the container (and therefore the single-node cluster):

   docker stop roach-single
   

   docker rm roach-single
   

2. If you do not plan to restart the cluster, you can also remove the Docker volume that contains the cluster's data:

   docker volume rm roach-single
   

What's next?

• Learn more about CockroachDB SQL and the built-in SQL client
• Install the client driver for your preferred language
• Build an app with CockroachDB
• Further explore CockroachDB capabilities like fault tolerance and automated repair, multi-region performance, serializable transactions, and JSON support

Was this helpful?