How to monitor long running scripts with pv

[matthias-van-woensel]

Goal

Sometimes you want to run a long running process without feedback. PV is a tool that allows you to do just that.

Example

• Running a mysqldump
  mysqldump <yourparams> | pv --progress --size 100m > /tmp/dump.sql
• Importing a mysql dump
  pv dump.sql | mysql <yourparams>
• Copying a large file
  pv /originallocation/largefile.bin > /otherlocation/largefile.bin
• Create a zip file
  pv largefile.txt | zip > largefile.zip
• Create a tar archive
  tar -czf - ./foldertotar | (pv -p --timer --rate --bytes > backup.tgz)

Problem

Sounds cool, but pv isn’t available by default.

Solution

Luckily, there is an easy solution.

Simply add the pv package in your .platform.app.yaml file. This will automatically install pv. So you can use it.

dependencies:
  nodejs: # Specify one NPM package per line.
    pv: '~1.0.1'

Note: you can do this in any container type. PHP, nodejs, golang, …

This works because there is a nodejs package that does the heavy lifting for us.

For more information on dependencies, check our documentation

[julix]

Hi @matthias-van-woensel,

I’d like to import a large db, something like pv latest--dump.sql | $(drush sql-connect)

Seems to have installed correctly as pv -v displays v1.0.1 and (as @travis.raup pointed out) cat /dev/urandom | pv --size 100MB > /dev/null also seems to work.

However, most things don’t - including the pv --help command:

They just get stuck (as per screenshot above). In the case of db import commands wrapped with PV - they still do their job, they just don’t update the cli (i.e. the pv part doesn’t work)

Opened a support ticket too, but figured you might know what to try?

Thanks!
Julian

[julix]

Sharing the results of my support request here:

William David Friday at 14:51

Hi Julian,

I set up pv in my own environment and replicated the behavior you saw. Taking a look at the code it looks like the version of pv that’s installed with node differs a bit from the version you might get if someone where to install it as a linux package. For instance, the “help” option just seems to not be built into this version.

In terms of monitoring database imports, it seems to work just by cat ing the file through it rather than invoking it directly, e.g. something like:

web@app.0:/tmp$ cat dump.sql | pv | mysql -h database.internal -u main -p main
Enter password: 
 100.00% | 0 ETA | 7.5MB Transferred | 1.2MB/s

Try that out and see if it helps?

Regards,
-Bill

Edit: confirmed that it works! Thank you, William David!