General Principles

In the automotive industry a design or manufacturing fault can cost millions in warranty and lost sales. As a result it has a highly-developed root cause analysis process designed to find, verify and fix problems. Just as with a full imaging system, automotive systems are complex and trust me, some of the root causes are unbelievably obscure. One of the processes used by the automotive, medical and defense industries is called 8D. The principles behind this process are equally applicable to any problem solving effort. These can be simplified to:

1 Record the symptom with as much detail as possible; it is a common mistake to interpret the symptom. “I have flu” is not a symptom but “I have nausea and a temperature” are.

2 Record when the symptom occurs and when it does not; what distinguishes those times when the problem occurs? This may be time, a different sensor or software version or a change in the “environment” for example. If the software records log files, keep them safe for later diagnostics.

3 Brainstorm the possible root causes; this is where you need system knowledge. An Internet search or the information in the tables may help. In all likelihood someone, somewhere, has had the same issue.

4 Evaluate the root causes and potential fixes; this may take the form of substitution (i.e. change cables, driver version, power supply, revert back to a working configuration) or an experiment to isolate variables and a compatibility check with the symptom. This last point is very powerful. Engineers talk about special and common causes. If there is a sudden problem in a fixed configuration that has otherwise been behaving itself for months, the issue is unlikely to be a design issue, or software. A sudden problem must be caused by a sudden change, like a hardware failure, software corruption or an “environment” change.

5 Verify the root cause and the potential fixes; many issues are intermittent and it is very easy to make a change which seemingly fixes the problem. For instance, swapping an electronic module in a system may make the problem go away, when it is the cleaning action of breaking and making the connection that actually fixes the issue. The solution is very simple: turn the problem off, on and off again with your “fix”. In the above example, plugging in the “broken” module also fixes the problem!

Problem solving is part of the hobby but it can be tiresome when you have the first clear night in a month and the system refuses to play nicely. Fixing a problem also provides an opportunity to prevent it happening again. This may take the form of regular mount maintenance, checking cables for connectivity and the old truism, “if it ain’t broke, don’t fix it”!

Hardware, Software and Customer Failures

So, which part of the system do you think is the most unreliable? Most likely you, or “customer error”, and with the complexity of ever-changing technology it is not surprising. The most complex problems to identify and fix are interactions between hardware and software, caused by poor design robustness (the system works in ideal circumstances but is intolerant to small changes in its environment) or device driver issues. A common example are the many issues caused by inexpensive USB hubs. They all seem to work in an office environment but refuse to operate reliably in a cold observatory. In this case, not only are some chip-sets more fault-tolerant than others, the surrounding analog components change their properties in cold conditions and the electrical performance suffers. The same applies to the lubricants used in some telescope mounts. Apparently unrelated changes to other parts of the computer operating system, a longer USB cable or interruptions caused by scheduled operating system maintenance may also be the culprit. It is a wonder that anything ever works!

symptom	possible root cause	notes
USB device disconnects	cable too long	may work when warm but fail when cold
(often as it cools down)	dew on connectors	protect with cloth over connectors in dewy
	too many daisy-chain hubs insufficient power chip-set or hardware clock frequency	restrict sequential USB hubs power overload on hub after initial connection some hardware is just not up to the job: look for hubs that use the NEC chip-set
	intermittent power during slew	use locking DC connectors where possible
device does not connect	also includes disconnect root causes above	try turning off / on or connect / disconnect cable
	ground offset (often seen with switch-mode DC power supplies) insufficient power	check to see if floating DC supplies have ground reference and check system grounding check USB hub power supply and cable type and are
	ASCOM driver issue wrong COM port wrong driver	reload / repair ASCOM drivers use device manager to confirm COM ports check hardware driver is up to date
USB serial failure	adaptors are not all created equal!	try one with FTDI / Prolific chip-set / Keyspan
broad stripes on any image	slow or interrupted USB interface	often seen on slow netbook computers
fine stripes on any image	camera clock stability within CCD	potentially curable with a firmware update
interference pattern on image	power supply noise during image download radiated or conducted radio frequency interference	check to see if CCD cooling is disabled during image download isolate power supply, apply ferrite clamps and shielding, check grounding and cable routing
dark frame evenness	CCD sensor issue	check with manufacturer what is considered normal, before claiming warranty repair
	light leak in hardware	confirm by exposing with lens cap on
dark frame evenness	light leak (IR transparency)	use metal lens caps where possible
exposure evenness	light leaks	check around filter wheel and OAG housing
	blooming around edges	possible CCD issue, or uneven cooling
	flare from nearby light source	extend the dew shield, remove the light
	light pollution near horizon	use light pollution filter or narrowband exposure to confirm
stars vertically “fragmented”	progressive scan CCD line-order is wrong	check advanced driver settings to swap line order
elongated stars (RA axis)	wrong tracking rate	is the sidereal rate set?
	autoguider issues (see autoguiding)
	periodic error (unguided)	check PE with utilities and use PEC
	refraction (at low altitude)	does mount support refraction compensation?
	exposure before post-dither settle	increase settle time or lower pixel error threshold
	poor polar alignment (unguided)	RA drift can also occur in specific cases
elongated stars (radial)	no field-flattener	insert a compatible field-flattener
	wrong sensor spacing to flattener	use a tool like CCDlnspector to confirm optimum
elongated stars (tangential)	field rotation from polar misalignment	seen most during long exposures and at high DEC
elongated stars (DEC axis)	drift due to polar misalignment	check your alignment process / drift align
	autoguider issues (see autoguiding)	backlash, drift, stiction, min. move set too high
	exposure before post-dither settles	increase settle time or lower pixel error limit
elongated stars (any axis)	guider output disabled	check guider controls and ST4 cable connection
	guider locked onto hot pixel	use dark frame or bad PixelMath calibration
	temporary clouds	stop buying new equipment!
	tripod movement (soft ground)	place legs on broad platform or deep spikes
out of focus (center)	worse over time	possible focus drift with thermal contraction
	worse with some filters	adjust focus for each filter position
	all the time	use autofocus tool or HFD to confirm best focus
	focuser mechanism slip	consider motorized rack and pinion focuser
out of focus (gradient)	sensor not square on	use laser jig to confirm sensor alignment
	sensor not square on	check focuser tube for play / sag
out of focus (corners)	field flatness	check sensor spacing to field-flattener
poor goto accuracy	poor polar alignment
	inaccurate time and location setting	check time zones and daylight saving settings
	telescope not synced / homed	often required to set a known position
	axis clutches slipping	check DEC / RA clutches are tight
star distortion with guiding	overcorrection from instability	check guider graph to confirm
(seen in image and also in guider graph)	overcorrection from incorrect calibration	check guider calibration with theoretical value
	overcorrection from stiction	possible with DEC axis, when moving from stationary
	overcorrection from high aggression	lower aggression setting
	maximum move set too low	set maximum move to 1 second
	guide rate set too low	increase guide rate by 25% and try again
	guide output inaccurate (seeing)	try increasing exposure to 5 seconds
	guide output inaccurate (seeing)	try binning to increase sensor SNR
	guide output inaccurate (flexure)	use OAG, tighten fasteners, lock mirrors
	constant guide error (no corrections)	lower minimum move setting in guider soft-ware
	constant guide error (with corrections)	backlash (DEC)
	slow to correct error (on graph)	increase max move, aggression or guide rate
star trails (graph good)	differential flexure	check rigidity of both optical/camera systems
star trails (sudden)	mount stops moving	check mount slew limits
	mount stops moving	tracking set to “off” accidentally?
star trails (after flip)	autoguider has lost guide star	re-acquire guide star and restart sequence
	system is applying RA in wrong direction	confirm settings to change RA guider polarity after meridian flip
star trails (stuttered)	wind or disturbance	check the cat is not sitting on the telescope (yes really)
	cable snag	route cables in polyester mesh sleeving and look for snag points
dumbbell stars (DEC)	DEC backlash is causing two-positions	tune mechanical backlash or use backlash compensation in software
dumbbell stars (any axis)	autoguider locks onto adjacent star	select guide star that is isolated from nearby stars
dumbbell star (RA)	DEC axis bearing preload	occurs when DEC guider changes polarity
small diffraction spikes	sometimes from lens spacers or micro-lens	seen on brighter stars, sorry, it’s physics
halo around bright stars	internal filter reflections	often in range of 30–60 pixels
	internal sensor reflections	often in range of 70–140 pixels
plate-solve fails	insufficient exposure / stars	longer exposure or choose different RA / DEC
	pixel scale estimate is wrong	check estimate for binning level used for exposure
	estimated position is wrong	check estimate, expand search area
	no estimates is available	use blind solve or all-sky solve (astrometry.net)
autofocus fails	star is too bright or dim	locate better star / change autofocus exposure
	optical parameters incorrect	check focal ratio and step size
	autofocus locks onto hot pixel	use dark frame calibration or select star in sub-frame manually
	focus tube does not move	check mechanical and electrical systems
inaccurate autofocus	too few sampling points	need minimum of 3 points either side of focus to establish V-curve
	over / under exposure	bright / dim stars are difficult to focus
	tracking issues during exposure	tracking problems distort HFD / FWHM measurement
	stars are too dim	try multi-star sampling to improve robustness of measurement
	focuser backlash	enable backlash compensation for moves that travel towards the ground
	autofocus through wrong filter bad seeing	develop strategy for autofocus and filter changes increase sampling per position to reduce effect
guider calibration fails	insufficient movement, low rate	increase calibration time, move to smaller DEC
	does not move / output disabled	hardware or control failure, check cables
	locks onto hot pixel for calibration	calibrate guider exposures and use filter / binning / sub-frame / dark pixel map or dark frame
	star moves off image	choose guide star away from edge of frame
	small RA movement	try calibrating at lower DEC setting (see guiding chapter on compensation pros/cons)
	lost star (poor SNR)	increase exposure, check focus, check for guide scope condensation
	bad calibration accuracy	seen in some cases where PE is excessive (100″)

fig.1 The most annoying problems are those that occur during image acquisition. The following list of common issues and possible causes may be of some use, many of which 1 have experienced at some time. These are “starters for ten” and are best considered in context to the occurrence of the issue and in response to the question, “What has changed”? Processing issues are not listed here since, although they are equally annoying, time is on your side to re-process the data to overcome the issue. Some visual clues, seen in images and software graphs, are shown in figs.2–9. (fig.1 is continued on the next three pages.)

fig.2 Bias frame downloaded on a slow USB connection (with a medium stretch applied). In this case, it was a direct USB 2.0 connection on a Netbook computer. Changing to a faster laptop fixed this issue.

fig.3 Over-correction during autoguiding (the guide rate was accidentally doubled after calibration) causing oscillation. The guider output is disabled at 220 seconds and the oscillation immediately stops.

fig.4 Subtly different to fig.3, multiple (DEC) corrections within each oscillation rule out over-correction from autoguider. In this case, it is a complex interaction within the mount firmware between encoders and autoguider inputs.

fig.5 Dumbbell stars, in this case caused by DEC backlash. (The camera is at 30° angle to DEC axis.)

fig.6 Field curvature in a crop from the top left corner. (Stars are perfectly round in center of image.)

fig.7 Lodestar image (unbinned), which has an incorrect line-order setting in its ASCOM driver.

fig.8 Maxim autoguider calibration image reveals mount backlash issue, as star does not return to starting position.

fig.9 Stretched master bias. CCD internal clock issue, fixed with a firmware update. Very difficult to calibrate out.

fig.10 Tracking oscillation, not drift. (Short exposure and a small hot spot in the middle of each line.)