Synchronisation

Heron, as an event based system, running over multiple processes and machines, does not rely on a single clock to generate synchronised behaviour. Yet, as an experiment pipeline generator, it needs to allow users to fully time match the different events and behaviours spawned by the different Nodes. To do this, Heron uses a number of logging systems, which, when combined, offer the required time matching capacity. The individual logging systems have been described in the Debugging section for the logging system keeping information on the communication between Nodes and the The Saving State System section for the system keeping information on the internal activity of each Node.

In what follows we will describe how those two systems can be used in conjunction and together with some external (hardware based) logging for the case of multiple machines, can generate a full record of the timing of every event that a Heron pipeline generates.

Hardware

Within a single machine, Heron timestamps all logged events using the machine’s clock. This of course breaks down when the pipeline includes Nodes running in multiple machines. The only way to time match events that are generated in different machines (at the required accuracy of an experimental setup) is to have an external data acquisition (DAQ) device that will register the timing of different events that happen on separate machines. Such a DAQ should save its data through a Heron Node running in the Heron GUI machine and have the saved data packets timestamped through Heron’s logging system.

Software

A Node can use Heron’s Node Save State System (see The Saving State System) to record information available to the worker process every time the worker function runs (for Transforms and Sinks) or goes around once in its infinite loop (for Sources). That allows the worker process to register the time and index of the packet it generates together with whatever other information it needs to save (the timestamping happens at the moment the savenodestate_update_substate_df function is called in the worker function).

The worker process of a Transform or a Sink though has no information about the packet that triggered the worker function, coming from a left connected Node. This information is saved in the Node’s com process log file ( described in Debugging). There, Heron saves both the timestamp and the index of the incoming packet from the left Node and the corresponding timestamp and the index of the packet the current worker function generates. In this way one can connect which packet from a left Node has triggered the production of which packet from a right Node.

Examples

To make the above more concrete lets give here two examples of simple pipelines and how one would go about time matching their outputs.

Within machine (saving a 120fps camera)

One of Heron’s Nodes is a Spinnaker Camera. Some of the FLIR cameras this Node can capture can reach large frames per second (FPS) speeds, too high for a standard Save CV2 Video Node to be fast enough to save appropriately. That is why within the Vision repo there is a Save FFMPEG Video Node.

Let’s assume we have a pipeline where a 120fps FLIR camera is captured by a Spinnaker Node and this feeds into a Save FFMPEG Video Node. Now, there are three possible places where frames can be lost:

  1. In the capturing from the camera to the Spinnaker Node.

  2. In the transfer from the Spinnaker to the FFMPEG Node.

  3. In the save into the video done in the FFMPEG Node.

Let’s also assume that our application does not require 100% perfect capture (i.e. zero dropped frames from camera to video file) but (as in most experimental cases) needs to allow the user to know exactly which frames have been dropped where and to be able to assign the real time a frame has been captured by the camera sensor to the corresponding video frame saved in the FFMPEG generated video file. They would also need to be able to assign every frame in the final video to any other event in the Heron pipline (e.g. a press of a button). In order to achieve this the user should:

  1. Assign a save directory to the “Save the Node State to director:” entry of the secondary window of the Spinnaker Node.

  2. Assign a log filename in the “Logfile or Verbosity Level” entry of the secondary window of the Save FFMPEG Video Node.

The Spinnaker Node saves the id and time of each captured frame as specified from the camera’s hardware together with the corresponding index and timestamp of the Node’s loop iteration. This way, any frames generated by the camera but are not captured by the Node become obvious.

The logfile generated by the FFMPEG Node’s com process defines the index and timestamp this process assigns to every packet (frame) that reaches it from the Spinnaker Node. Here, any frame that doesn’t make it across the Nodes will become obvious but more importantly this log file will generate a one to one correspondence between the indices of the frames that pass through as they are generated by the Spinnaker Node and the indices generated by the FFMPEG Node. Finally the FFMPEG Node uses an FFMPEG pipeline to write every receiving frame into an encoded video file. This needs to be optimised (in the pipeline the FFMPEG Node defines) so that the number of frames in the final video should be equal to the number of indices in the log file of the FFMPEG Node (that means the 3. above should never happen).

Following the above procedure one can create a one to one connection between any video frame and the time the computer captured that frame from the camera. They can also use the computer timestamp of any frame (the timestamp that marks when the frame was captured by the Spinnaker Node) to assign every video frame to any other timestamped event in Heron’s pipeline as long as this happened in a Node running in the same Heron GUI machine.

Across Machines (time matching frames of different cameras)

Now let’s assume that in the above described Heron pipeline there is also an Arducam camera Node running on a Raspberry Pi connected to the same LAN as the Heron GUI computer. The goal here is to be able to assign every frame from the FLIR camera saved onto the FFMPEG generated video to every frame captured by the Arducam camera and saved to the Arducam video. Here a user would need to again provide a save directory to the Arducam Node’s secondary window. This way the Arducam Node would save the timestamp and the index of the captured frame, which, because the saving of the video happens within the same Arducam Node, would correspond to the equivalent saved frame in the video.

Yet that would not be enough to time match the frames of the two videos to each other, since the timestamps for the Arducam frames are generated by a different computer to the ones assigned to the FFMPEG Video. The only way around this would be for the user to have a DAQ that captures hardware triggers for both the FLIR camera and the Arducam one and save those through a separate Node (e.g. the NIDAX Node). At the same time the Arducam Node can capture a TTL pulse that comes into the Ras Pi’s GPIO and save its index and time onto the Save State dataframe together with the index of the captured frame. This saved TTL pulse is the same trigger that triggers the Arducam camera and is also saved in the DAQ. This way the DAQ becomes the central clock onto which the camera captured frames from both camera systems get registered. Through this common clock and the use of Heron’s Save State dataframes and com process log files, all events on both computers can be time matched to any other event.