https://mg.pov.lt/blog/tags/buildbot.html Recent content in Buildbot on Random notes from mg Hugo -- gohugo.io en-us marius@gedmin.as (Marius Gedminas) marius@gedmin.as (Marius Gedminas) Copyright © 2004–2020 Marius Gedminas Fri, 15 May 2009 15:33:26 +0300 https://mg.pov.lt/blog/hardy-nfs-sighup.html Fri, 15 May 2009 15:33:26 +0300 marius@gedmin.as (Marius Gedminas) https://mg.pov.lt/blog/hardy-nfs-sighup.html <p><strong>Update</strong>: The story continues, but solution is not in sight yet.</p> <p>I upgraded a buildbot slave to Ubuntu 8.04 (Hardy) recently and now I'm getting a strange intermittent failure: sometimes <tt>cp -r /local/dir /nfs/mounted/dir</tt> fails ("process killed by signal 1", i.e. SIGHUP).</p> <p>I wonder if NFS is relevant or incidental to the issue?</p> <p>Google finds <a href="http://osdir.com/ml/python.buildbot.devel/2005-07/msg00000.html">an old thread from 2005</a>, with a workaround (usepty=False), but I'd like to understand the problem before applying random fixes.</p> <p>So far three different build steps doing <tt>cp -r</tt> have failed during 10 days. I've now changed them all to <tt>cp -rv</tt>, so I can at least see if the failure is in the middle of the copy or at the end, if it fails again.</p> <p><strong>Update</strong>: so far 4 build steps have failed on 6 separate occasions:</p> <!-- ./ivija-coverage/754-log-cp-stdio ./ivija-coverage/757-log-cp-stdio ./ivija-coverage/757-log-cp_2-stdio ./ivija-coverage/770-log-rm_2-stdio ./ivija-coverage/773-log-rm_2-stdio ./ivija-docs/342-log-cp-stdio --> <pre> May 5 02:31: cp -r local-dir1 nfs-mounted-dir1 <!-- ivija-coverage cp --> May 6 02:31: cp -r local-dir1 nfs-mounted-dir1 <!-- ivija-coverage cp --> May 6 04:33: cp -r local-dir2 nfs-mounted-dir2 <!-- ivija-coverage cp_2 --> May 15 02:00: cp -r local-dir3 nfs-mounted-dir3 <!-- ivija-docs cp --> May 17 04:32: rm -rf nfs-mounted-dir4 <!-- ivija-coverage rm_2 --> May 20 04:31: rm -rf nfs-mounted-dir4 <!-- ivija-coverage rm_2 --> </pre> <p>I see no particular correlation between step duration and results, e.g. the rm -rf step usually takes between 2.2 and 4.6 seconds. The two SIGHUPs happened after 2.4 seconds. </p> <!-- cd /var/lib/buildbot/masters/ivija python import pickle, pprint pp = pprint.pprint coverage_jobs = [pickle.load(file('ivija-coverage/' + str(n))) for n in range(750, 774)] docs_jobs = [pickle.load(file('ivija-docs/' + str(n))) for n in range(300, 349)] jjobs = docs_jobs pp(['%s %.1f %s' % (s.name, s.finished - s.started, s.results) for s in (b.steps[[ss.name for ss in jobs[-1].steps].index('cp')] for b in jobs) if s.finished]) rm_2 takes between 2.2 and 4.6 seconds. The two failures were at 2.4 seconds. cp_2 takes between 6.2 and 19.2 seconds. The one failure was after 7.2 seconds. cp takes between 3.5 and 15.7 seconds. The two failures were after 3.8 and 4.1 seconds. ivija-docs cp takes between 0.8 and 4.1 seconds. The failure was after 1.6 seconds. --> <p>They all make no output. When I changed the cp steps and added a -v, they stopped failing, but that could be just a coincidence.</p> <p>We're having an email conversation with Jean-Paul Calderone ("exarkun") about the possibility of this being PTY-related, with no clear resolution so far.</p> <p>And, hey, now this blog supports comments ;)</p>