The RandomTriggerCandidateMaker module broke in fddaq-v4.4.2, and I suspect that the problem was caused by a change in the TimestampEstimator

[bieryAtFnal]

I will post instructions for reproducing the problem in a moment. Those instructions will have two parts. The first part will set up a software area based on fddaq-v4.4.4, and they will demonstrate that with a fddaq-v4.4.4 software area using the RTCM

1. there are error messages in the Trigger log file complaining about "rtcm_input: Unable to push within timeout period (timeout period was 10 milliseconds)"
2. there appear to be TCs generated by the RTCM at a very high rate

The second part of the instructions will ideally be run immediately after the first part, and they update the TimestampEstimator code in the utilities repo to be the versions that were part of fddaq-v4.4.1 (coredaq-v4.5.1). With those files rolled back, the system runs fine, and RTCM TCs are created at 1 Hz, as expected.

[bieryAtFnal]

Here are the two parts of the instructions to demonstrate the problem:

DATE_PREFIX=`date '+%d%b'`
TIME_SUFFIX=`date '+%H%M'`

source /cvmfs/dunedaq.opensciencegrid.org/setup_dunedaq.sh
setup_dbt fddaq-v4.4.4
dbt-create fddaq-v4.4.4-a9 ${DATE_PREFIX}FDv4.4.4${TIME_SUFFIX}
cd ${DATE_PREFIX}FDv4.4.4${TIME_SUFFIX}/sourcecode

git clone https://github.com/DUNE-DAQ/daqsystemtest.git -b fddaq-v4.4.4
git clone https://github.com/DUNE-DAQ/trigger.git -b coredaq-v4.5.4
git clone https://github.com/DUNE-DAQ/utilities.git -b coredaq-v4.5.4
cd ..

dbt-workarea-env
dbt-build -j 20
dbt-workarea-env

mkdir rundir
cd rundir

cat <<EOF1 > daqconf.json
{
  "boot": {
    "use_connectivity_service": true,
    "start_connectivity_service": true,
    "connectivity_service_host": "localhost",
    "connectivity_service_port": 15432
  }, 
  "daq_common": {
    "data_rate_slowdown_factor": 1
  },
  "detector": {
    "clock_speed_hz": 62500000
  },
  "readout": {
    "use_fake_cards": true,
    "default_data_file": "asset://?label=WIBEth&subsystem=readout"
  },
  "trigger": {
    "mlt_merge_overlapping_tcs": false,
    "use_random_maker": true,
    "rtcm_timestamp_method": "kTimeSync",
    "rtcm_time_distribution": "kUniform",
    "rtcm_trigger_interval_ticks": 62500000,
    "mlt_use_readout_map": true,
      "mlt_td_readout_map": [
          {
              "candidate_type": 4,
              "time_before": 1000,
              "time_after": 1000
          }
      ]
  },
  "dataflow": {
    "apps": [ { "app_name": "dataflow0" } ]
  },
  "hsi": {
    "use_fake_hsi": false,    
    "use_timing_hsi": false,    
    "random_trigger_rate_hz": 1.0
  }
}
EOF1

cat <<EOF2 > dro_map.json
[
    {
        "src_id": 100,
        "geo_id": {
            "det_id": 3,
            "crate_id": 1,
            "slot_id": 0,
            "stream_id": 0
        },
        "kind": "eth",
        "parameters": {
            "protocol": "udp",
            "mode": "fix_rate",
            "rx_iface": 0,
            "rx_host": "localhost",
            "rx_pcie_dev": "0000:00:00.0",
            "rx_mac": "00:00:00:00:00:00",
            "rx_ip": "0.0.0.0",
            "tx_host": "localhost",
            "tx_mac": "00:00:00:00:00:00",
            "tx_ip": "0.0.0.0"
        }
    },
    {
        "src_id": 101,
        "geo_id": {
            "det_id": 3,
            "crate_id": 1,
            "slot_id": 0,
            "stream_id": 1
        },
        "kind": "eth",
        "parameters": {
            "protocol": "udp",
            "mode": "fix_rate",
            "rx_iface": 0,
            "rx_host": "localhost",
            "rx_pcie_dev": "0000:00:00.0",
            "rx_mac": "00:00:00:00:00:00",
            "rx_ip": "0.0.0.0",
            "tx_host": "localhost",
            "tx_mac": "00:00:00:00:00:00",
            "tx_ip": "0.0.0.0"
        }
    }
]
EOF2

fddaqconf_gen -c ./daqconf.json --detector-readout-map-file ./dro_map.json mdapp_RTCM

cd ..
export TRACE_FILE=`pwd`/log/${USER}_dunedaq.trace
tinfo
tonS -n RandomTriggerCandidateMaker DEBUG+10

cd rundir
nanorc --partition-number 2 mdapp_RTCM ${USER}-test boot conf start_run 101 wait 20 stop_run scrap terminate

egrep 'ERROR|WARNING' log_*.txt

sleep 6

grep RTCM log_trigger*.txt | tail

and

# expects to be run in $DBT_AREA_ROOT/rundir after "part1"...

mkdir backup
mv log_*.txt backup/.

cd ../sourcecode/utilities/
git checkout coredaq-v4.5.1 include/utilities/TimestampEstimator.hpp include/utilities/detail/TimestampEstimator.hxx src/TimestampEstimator.cpp
cd ../../

dbt-build -j 12 --clean
dbt-workarea-env

cd rundir/
nanorc --partition-number 2 mdapp_RTCM ${USER}-test boot conf start_run 201 wait 20 stop_run scrap terminate
egrep 'ERROR|WARNING' log_*.txt
ls -alF

[bieryAtFnal]

For reference, I'll add that my investigation of this problem began with a message that Michal Rigan posted on the daq-integration Slack channel mentioning a problem when running the 3-trigger-type sample daqconf.json file in the fddaq-v4.4.4 software area instructions.