Docker-composer.yml: services.farmer.depends_on contains an invalid type, it should be an array

Hi. I’m new to Docker and I’m getting the following error when running docker-compose up -d:

ERROR: The Compose file './docker-compose.yml' is invalid because:

services.farmer.depends_on contains an invalid type, it should be an array

From the v3 documentation it seems there is a new syntax and the following caveat:

There are several things to be aware of when using depends_on :

  • depends_on does not wait for db and redis to be “ready” before starting web - only until they have been started. If you need to wait for a service to be ready, see Controlling startup order for more on this problem and strategies for solving it.
  • The depends_on option is ignored when deploying a stack in swarm mode with a version 3 Compose file.

If I downgrade the version to 2 I get the following:

Creating network "subspace_default" with the default driver
Creating volume "subspace_node-data" with default driver
Creating volume "subspace_farmer-data" with default driver
Pulling node (ghcr.io/subspace/node:snapshot-2022-may-03)...
snapshot-2022-may-03: Pulling from subspace/node
8e5c1b329fe3: Pull complete
4d2b9e83b5af: Pull complete
4e8e3a4ce61d: Pull complete
5fa6fe89005e: Pull complete
Digest: sha256:3d5c0def57b8d5a91c599a6fc22d0a3bbcca318ec2d6ba6243c05b1d6f4b115e
Status: Downloaded newer image for ghcr.io/subspace/node:snapshot-2022-may-03
Pulling farmer (ghcr.io/subspace/farmer:snapshot-2022-may-03)...
snapshot-2022-may-03: Pulling from subspace/farmer
8e5c1b329fe3: Already exists
865967514474: Pull complete
510061880758: Pull complete
Digest: sha256:e13586625adb3eaa3e84b6c62a452e9afa0217aeea86e828a757be3f3c62bb21
Status: Downloaded newer image for ghcr.io/subspace/farmer:snapshot-2022-may-03
Creating subspace_node_1 ... done

ERROR: for farmer  Container "8cffceccc881" is unhealthy.
ERROR: Encountered errors while bringing up the project.

What do you advise? Thanks

1 Like

Would you mind posting your docker-compose.yml?

Sure

version: "3.7"
services:
  node:
    # Replace `snapshot-DATE` with latest release (like `snapshot-2022-apr-29`)
    image: ghcr.io/subspace/node:snapshot-2022-may-03
    volumes:
# Instead of specifying volume (which will store data in `/var/lib/docker`), you can
# alternatively specify path to the directory where files will be stored, just make
# sure everyone is allowed to write there
#      - node-data:/var/subspace:rw
      - /media/subspace/node:/var/subspace:rw
    ports:
# If port 30333 is already occupied by another Substrate-based node, replace all
# occurrences of `30333` in this file with another value
      - "0.0.0.0:30333:30333"
    restart: unless-stopped
    command: [
      "--chain", "testnet",
      "--base-path", "/var/subspace",
      "--execution", "wasm",
      "--pruning", "1024",
      "--keep-blocks", "1024",
      "--port", "30333",
      "--rpc-cors", "all",
      "--rpc-methods", "safe",
      "--unsafe-ws-external",
      "--validator",
# Replace `INSERT_YOUR_ID` with your node ID (will be shown in telemetry)
      "--name", "[redacted]"
    ]
    healthcheck:
      timeout: 5s
# If node setup takes longer then expected, you want to increase `interval` and `retries` number.
      interval: 30s
      retries: 5

  farmer:
    depends_on:
      node:
        condition: service_healthy
# Replace `snapshot-DATE` with latest release (like `snapshot-2022-apr-29`)
    image: ghcr.io/subspace/farmer:snapshot-2022-may-03
# Un-comment following 2 lines to unlock farmer's RPC
#    ports:
#      - "127.0.0.1:9955:9955"
# Instead of specifying volume (which will store data in `/var/lib/docker`), you can
# alternatively specify path to the directory where files will be stored, just make
# sure everyone is allowed to write there
    volumes:
#      - farmer-data:/var/subspace:rw
      - /media/subspace/farmer:/var/subspace:rw
    restart: unless-stopped
    command: [
      "--base-path", "/var/subspace",
      "farm",
      "--node-rpc-url", "ws://node:9944",
      "--ws-server-listen-addr", "0.0.0.0:9955",
# Replace `WALLET_ADDRESS` with your Polkadot.js wallet address
      "--reward-address", "[Redacted]",
# Replace `PLOT_SIZE` with plot size in gigabytes or terabytes, for instance 100G or 2T (but leave at least 10G of disk space for node)
      "--plot-size", "1T"
    ]
volumes:
  node-data:
  farmer-data:

Looks like you may have used outdated documentation, here is the proper docs for this release subspace/farming.md at 94eaeb446f83aa2ac68abd504b2f79bbc7964220 · subspace/subspace · GitHub

Thanks. I’m getting the same result though. docker-compose still complains about the syntax if I set version to “3.x”. This is the error I’m getting in the docker-compose logs if I downgrade to “2.x”:

node_1 | Error: SubstrateService(Other("Failed to build a full subspace node: Permission denied (os error 13)"))

I’m running docker-compose vs 1.25.0 and Ubuntu 20.04.4 LTS, tried with sudo and without sudo. The user was added to the docker group. I’ve noticed it creates the node folder as root.

Hmmm, it works just fine for me on 3.7, did you ensure to run docker-compose down -v to revert failed attempts? If you did not do this you could have some running containers causing conflicts.

Verify Docker on Linux

I also saw that you mentioned you added the user to the docker group, did you follow all of the Linux post-installation steps for Docker Engine | Docker Docs?

Particularly #4

  1. Verify that you can run docker commands without sudo .

$ docker run hello-world

This command downloads a test image and runs it in a container. When the container runs, it prints a message and exits.If you initially ran Docker CLI commands using sudo before adding your user to the docker group, you may see the following error, which indicates that your ~/.docker/ directory was created with incorrect permissions due to the sudo commands.

WARNING: Error loading config file: /home/user/.docker/config.json -
stat /home/user/.docker/config.json: permission denied

To fix this problem, either remove the ~/.docker/ directory (it is recreated automatically, but any custom settings are lost), or Ichange its ownership and permissions using the following commands:

$ sudo chown “$USER”:“$USER” /home/"USER"/.docker -R sudo chmod g+rwx “$HOME/.docker” -R

If you’ve verified that is working fine, it may be best to try cleaning up your docker installation a bit

Docker Clean Up

Best case

You can try running docker-compose down -v but it may fail in which case you can try below

Clean up Docker if docker-compose down -v is failing

Warning, this assumes you aren’t running other containers on this machine, and will wipe and remove all containers/volumes

  1. List all docker containers
    docker container ls

  2. The list will show all containers docker has on your machine, take note of the status, if any are running you will want to stop them
    docker container stop [container-name]

  3. Once all of the containers are stopped (or at least any that may be subspace related) we will prune the containers
    *This command will delete all stopped containers
    docker container prune -f

  4. We will then remove all of the volumes
    *This command will delete all volumes
    docker volume prune -f

Conclusion

We know this version supports 3.7 with no major issues (im using Ubuntu LTS 20.04 aswell), as such its likely some permission error with the directory/user that is being used by your docker installation.

Let me know if none of these work, im happy to continue to help troubleshoot :slight_smile:

Hey Justin. Thanks for your answer. I did complete all of the above, but still getting the same issue. I’ll stay on CLI for the time being and will try to figure this out as time permits.

Let me know if you have more troubleshooting ideas. One think that could be helpfull is to know who should be the owner of the node and farmer volumes created on startup? In my setup they are showing up as owned by root.

1 Like

The volume and the directory the volume is being mounted on should all be owned by the user, that may be part of our issue

Some progress in this front as I’ve installed docker from scratch on another machine and the syntax issue is gone, so there was some issue with my original installation.

Still I can’t get through the permission denied error. It happens only if I use bind mount instead of volumes .

The mount seems fine from docker inspect:

        "Mounts": [
            {
                "Type": "bind",
                "Source": "/media/subspace/node",
                "Destination": "/var/subspace",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            }

and the node data directory is created with the following user/permissions:

drwxr-xr-x 2 root root 4096 Jun 11 09:07 node

The log output:

2022-06-11 09:02:01 Subspace    
2022-06-11 09:02:01 ✌️  version 0.1.0-00670c9714949ec8f776e7f0467110945866457b    
2022-06-11 09:02:01 ❤️  by Subspace Labs <https://subspace.network>, 2021-2022    
2022-06-11 09:02:01 📋 Chain specification: Subspace Gemini 1    
2022-06-11 09:02:01 🏷  Node name: [redacted]   
2022-06-11 09:02:01 👤 Role: AUTHORITY    
2022-06-11 09:02:01 💾 Database: ParityDb at /var/subspace/chains/subspace_gemini_1b/paritydb/full    
2022-06-11 09:02:01 ⛓  Native runtime: subspace-1 (subspace-0.tx0.au0)    
Error: SubstrateService(Other("Failed to build a full subspace node: Sub(Keystore(Io(Os { code: 13, kind: PermissionDenied, message: \"Permission denied\" })))"))

actually ran into a similar issue on docker the other day. Did you by chance install docker with Snap?

No, with apt following the oficial guide.

Great answer to the initial issue on this post. Credit goes to Discord user Big Jim | counterpoint

This is caused by running an old version of Docker. Updating to the latest version fixes it. I do think instructions on how to do that would be helpful in the guide as this question seems to come up in the support and general chats quite frequently.

For anyone who encounters the depends_on contains an invalid type, it should be an array error when running docker compose it could well be that you’re not on the latest version of Docker Compose.

I managed to fall foul of this by running apt install docker-compose which installs v.1.25.0. This version does not parse version 3 docker-compose.yml files properly and cannot understand the (super useful) depends_on with condition features.

The shortest way to fix this appears to be to uninstall Docker Compose and re-install it from the official guide rather than from apt.
The below worked for me:

  • sudo apt remove --purge docker-compose
  • sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
  • sudo chmod +x /usr/local/bin/docker-compose

If all has worked you should get docker-compose version 1.29.2, build 5becea4c or similar from docker-compose --version and you can go ahead and pull/up. This is the official installation guide which I encourage you to at least skim before trusting some random on the internet Install Docker Compose | Docker Documentation

2 Likes

I would suggest sticking with the volume route if that worked, while bind mounts can be a little bit more performant, I do not believe they have been tested just yet and I do not have too much experience with them.

@nazar-pc any insight on this?

Yes I think this is what I did in the first install. Thanks for making it clear.

I would suggest sticking with the volume route if that worked

I would like to have the node/farmer data in an external drive independent of where docker data root is. I’ve imagined this would be a common requirement :thinking: So I’ll stick with CLI for now.

You don’t give any writes permission to anyone except root. Subspace node and farmer are running under reduced permissions nobody:nogroup. So either allow anyone to write data there or change the user under which containers are running.

1 Like

I was pretty sure I’ve tried changing permissions. It seems I didn’t :man_facepalming: It’s working fine now thanks!

1 Like