During the configuration process, you’ll be asked to enter your password.When connecting to a remote host over ssh, by configuring the DOCKER_HOST environment variable, or by using the docker context functionality and specifying the host= parameter, some Linux/Unix environments require additional setup because the docker CLI assumes that the host it is SSHing into has docker located in the PATH.Based on your OS, your DOCKERHOST is different. On Linux, it will simply be your localhost.$ docker run -it -rm -privileged -pid =host justincormack/nsenter1This gives you full access to the Docker VM. Echo 'PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin' > ~/.ssh/environmentSudo sh -c 'echo "PermitUserEnvironment PATH" > /private/etc/ssh/sshd_config'Describe the results you received: error during connect: Get command has exited with exit status 127, please make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=zsh:1: command not found: dockerI expected it to be relatively straight forward to provide an SSH host address for another machine running docker and that docker could find itself.I think it would significantly simplify using docker remotely across machines if docker didn't only rely on PATH but searched some common paths that Docker typically installs itself, such as /usr/local/bin/.Additional information you deem important (e.g. Issue happens only occasionally):Output of docker version: Client: Docker Engine - CommunityGitCommit: 269548fa27e0089a8b8278fc4fc781d7f65a939bGitCommit: ff819c7e9184c13b7c2607fe6c30ae19403a7affApp: Docker App (Docker Inc., v0.9.1-beta3)Buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)Network: bridge host ipvlan macvlan null overlayLog: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslogRuntimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2Containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939bRunc version: ff819c7e9184c13b7c2607fe6c30ae19403a7affID: BTMT:FQ3F:RCNB:IDF5:W7AP:7NRK:XSTI:7ERE:62JZ:3ODT:IARG:OQWCHTTPS Proxy: gateway.docker.internal:3129Additional environment details (AWS, VirtualBox, physical, etc.):This is all running on physical machines. I had the same problem with one flavor of Linux a little while ago on AWS, but I didn't note the details of which distro and version and don't remember. Docker run -d -rm -name nginx -p 80:80 nginx:1.10.3 I can access Nginx on port 80 on my Mac.It displays each container and its CPU usage, memory usage, and other resource information. In my case, running pidstat and mpstat was sufficient to discover the cause of my slow builds.One helpful Docker-specific tool is docker stats, which is a top-like tool for monitoring the status of running Docker containers. I recommend that you install these inside of your Docker containers to help with performance analysis. Netflix has an excellent post called Linux Performance Analysis in 60,000 Milliseconds which recommends some good tools. Basic observabilityBefore diving too deeply into perf or eBPF to look for performance problems, it helps to know where to start.
Privileged containers, in order to work. Most of its functionality require root access, i.e. This directory is accessible from within your VM, so you’ll want to run this command to start a shell and inspect it:Host$ docker run -it -rm -privileged -pid =host justincormack/nsenter1Container# cat /sys/fs/cgroup/cpu,cpuacct/docker/./cpu.statPerf is a really powerful tool that provides visibility into a lot of different events on a Linux system. This is a good way to see if performance issues are due to container issues (cgroup throttling) or if they’re due to host issues such as CPU saturation. Docker containers can be throttled by cgroup CPU limits, and when this happens a throttle counter will be incremented in /sys/fs/cgroup/cpu,cpuacct/docker//cpu.stat. Volume license serializer for office 2016 for mac kaceThese do not ship with the Docker Desktop for macOS VM, so I had to find a way to compile and install them which is the hardest part of getting BCC tools and bpftrace working.Fortunately, bpftrace has an informative guide on installing kernel headers which I used as a starting point to get eBPF working. EBPF tools such as BCC and bpftrace, especially for older Linux Kernels, rely on Linux kernel headers. BCC tools and bpftraceBCC tools and bpftrace provide a way to perform more in-depth performance analysis, but require a little bit more work to get up and running in the macOS Docker environment.In most cases, if you run Docker on a Linux host it shares the kernel with the Linux host. These make it really easy to visualize where your program is spending lots of time. An alternative would be to run docker exec into your running container, which would work well for something like a server.Host$ docker run -priviliged -v $PWD:perf_data:/perf_data -rm bashI recommend checking out CPU Flame Graphs, a blog post by Brendan Gregg, which outlines how to use perf output to make flamegraphs. This involves either manually running apt-get install linux-perf immediately after spawning a shell or adding it to your Dockerfile.The commonly used perf record command dumps data to disk, so I also recommend mounting a volume to persist data across container runs and to do analysis on your host machine after profiling is done.I also installed tmux so I could run the command causing the problem in one pane, and run perf record in the other pane. Docker Docker_Host Download The LinuxUnfortunately my kernel version didn’t have a release tag in the linuxkit repository, so I had to do a checkout and clone instead of downloading a tarball.I also didn’t want to have to send the Linux kernel source (which is around 1GB) to the Docker daemon for each build, so I decided to add the 4.19.121-linuxkit directory to. Download the linuxkit sourceHere is the command I ran on the host (my macOS laptop) to download the Linux kernel sources. I’m not sure if there is any difference between the two, but I used the linuxkit fork to be safe. Run the Docker container and prepare headersBCC looks for kernel headers in the /lib/modules/$(uname -r)/source and /lib/modules/$(uname -r)/build directories, which are typically symlinked to /usr/src/$(uname-r). These could probably be automated in an entrypoint.sh script. However, this also means that there are some manual steps to run after the container boots up. This would ensure that preparing the Linux headers would persist across container runs and the container would boot up fast. QuestionsAlthough I was able to figure out how to run eBPF, there were some things I couldn’t figure out. We ended up fixing the issue by installing a version of lxml which had a pre-built libxml2 wheel which didn’t require compiling.So I didn’t end up needing eBPF in the end, but I still went through with the installation process as an educational exercise. Running pidstat showed that gcc was responsible for using most of the CPU during the build, and from there I was able to narrow it down to a libxml2 build using top and ps. Mpstat -P ALL was showing that all 4 processors were running at around 90% during most of the build, but then at some point went down to 100% for 1 processor, indicating some bottleneck in the build process.Since the build was configured to install dependencies in parallel, this was very unexpected. I only needed to use pidstat and mpstat to figure the issue out.
0 Comments
Leave a Reply. |
AuthorCandice ArchivesCategories |