Update pipelines to identify and load UTF-8 encoding

[cha801p]

When handling data that is not encoded in UTF-8 and has special characters, special characters can sometimes cause issues, leading to failed fetchers and failure with data pre-ingestion.

Action Required: A strategy is required to handle the specific encoded data in pre-ingestion. Potential solutions include:

1. Identifying and handling non-UTF-8 encoded characters.
2. Implementing error handling strategies such as errors='ignore' or errors='replace' in the file reading process.

Related Issues

Issue: The dr10574-ho-avh-weekly job has been consistently failing over the last few weeks. Upon further investigation, the failure was traced to an issue in the fetcher. The following error was noted:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 1786: invalid start byte
Upon thorough investigation of the data, an issue was observed with certain characters. The problematic data involves latitude and longitude values, specifically:
Latitude and longitude are automatically calculated from verbatim grid reference. Unadjusted latitude: 42�35'S. Unadjusted longitude: 147�52'E.,-42.583333,147.866119,EPSG:4202,50,0.000278,AMG 55 571100 5287900

Issue: NSW AVH reported a similar issue with UTF-8 encoding.
Example: AVH Occurrence Record
Supplied value: (Sessé & DC. ex Moç.) Coville
Processed value: R.Br.
Furthermore, the data provider supplied the reformatted data file with UTF-8 encoding which was loaded on the production.

Ticket Reference: Support Ticket #205967
GIT issue #1073